[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:57:49 dist-test-slave-cvwc krb5kdc[29463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:57:49 dist-test-slave-cvwc krb5kdc[29463](info): set up 2 sockets
Aug 10 14:57:49 dist-test-slave-cvwc krb5kdc[29463](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 14:57:50.563382 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:57:50 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:57:50.682744 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42059
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:50.764348 29487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:50.764577 29487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:50.764607 29487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:50.765827 29487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:57:50.765870 29487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:50.765888 29487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:50.765903 29487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:57:50.765928 29487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:57:50.767477 29487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:50.767740 29487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:50.767966 29487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:50.773231 29501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:50.773236 29497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:50.773236 29498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:50.773515 29487 server_base.cc:1034] running on GCE node
I20260810 14:57:50.773794 29487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:50.774125 29487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:50.775281 29487 hybrid_clock.cc:648] HybridClock initialized: now 1786373870775255 us; error 32 us; skew 500 ppm
Aug 10 14:57:50 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:50.780339 29487 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:57:50.780764 29487 webserver.cc:460] Webserver started at http://127.28.193.254:46503/ using document root <none> and password file <none>
I20260810 14:57:50.780997 29487 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:50.781046 29487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:50.781183 29487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:50.782382 29487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "7f667f50f95b4cd6ad0a00b212030b82"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "142a08719c60e2c3a9c53f68971635cf"
server_key_iv: "0dd7d5c9b12862ddf5ab756a33d957ca"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.782716 29487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "7f667f50f95b4cd6ad0a00b212030b82"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "142a08719c60e2c3a9c53f68971635cf"
server_key_iv: "0dd7d5c9b12862ddf5ab756a33d957ca"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.784897 29487 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:50.786449 29510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:50.786736 29487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:50.786861 29487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "7f667f50f95b4cd6ad0a00b212030b82"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "142a08719c60e2c3a9c53f68971635cf"
server_key_iv: "0dd7d5c9b12862ddf5ab756a33d957ca"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.786934 29487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:50.814759 29487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:50.816777 29487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:50.816923 29487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:50.820966 29487 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:42059
I20260810 14:57:50.821004 29597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:42059 every 8 connection(s)
I20260810 14:57:50.821357 29487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:57:50.822374 29598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:50.824697 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82: Bootstrap starting.
I20260810 14:57:50.825294 29598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:50.825548 29598 log.cc:824] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:50.826301 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82: No bootstrap required, opened a new log
I20260810 14:57:50.827880 29598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } }
I20260810 14:57:50.828022 29598 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:50.828043 29598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f667f50f95b4cd6ad0a00b212030b82, State: Initialized, Role: FOLLOWER
I20260810 14:57:50.828154 29598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } }
I20260810 14:57:50.828223 29598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:57:50.828248 29598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:57:50.828275 29598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:50.828353 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29487
I20260810 14:57:50.828434 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:57:50.828754 29598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } }
I20260810 14:57:50.828836 29598 leader_election.cc:302] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f667f50f95b4cd6ad0a00b212030b82; no voters:
I20260810 14:57:50.828962 29598 leader_election.cc:288] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:57:50.829007 29602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:50.829031 29447 external_mini_cluster.cc:1428] Setting key 3e00225bb64ac8e983ef1542bd3c1fe5
I20260810 14:57:50.829123 29602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [term 1 LEADER]: Becoming Leader. State: Replica: 7f667f50f95b4cd6ad0a00b212030b82, State: Running, Role: LEADER
I20260810 14:57:50.829214 29598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:57:50.829193 29602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } }
I20260810 14:57:50.829527 29602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f667f50f95b4cd6ad0a00b212030b82. Latest consensus state: current_term: 1 leader_uuid: "7f667f50f95b4cd6ad0a00b212030b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } } }
I20260810 14:57:50.829591 29602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:50.829737 29603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f667f50f95b4cd6ad0a00b212030b82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f667f50f95b4cd6ad0a00b212030b82" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42059 } } }
I20260810 14:57:50.829829 29603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82 [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:50.830714 29614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:57:50.831451 29614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:57:50.833292 29614 catalog_manager.cc:1347] Generated new cluster ID: ecbf1444b0d24ebbbc74d13d4d195dcc
I20260810 14:57:50.833351 29614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:57:50 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:50.848094 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:50.830131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.28.193.254:42059)
0810 14:57:50.830566 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:50.830576 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:50.830912 (+ 336us) server_negotiation.cc:407] Connection header received
0810 14:57:50.833205 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:50.833213 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:50.833269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:50.833415 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:50.835664 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:50.837101 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:50.838689 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:50.839313 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:50.839436 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:50.843854 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:50.843858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:50.843859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:50.843871 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:50.845937 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:50.846439 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:50.846442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:50.846442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:50.846543 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:50.846821 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:50.846822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:50.846822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:50.847129 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:50.847242 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:50.847647 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 14:57:50.847839 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":76,"threads_started":1}
I20260810 14:57:50.860199 29614 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:57:50.860728 29614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:57:50.868837 29614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f667f50f95b4cd6ad0a00b212030b82: Generated new TSK 0
I20260810 14:57:50.869037 29614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:57:50.894676 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:50.978174 29637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:50.978379 29637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:50.978401 29637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:50.979538 29637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:50.979579 29637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:50.979604 29637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:57:50.981033 29637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:50.981226 29637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:50.981392 29637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:50.986181 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:50.986197 29648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:50.986294 29655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:50.986636 29637 server_base.cc:1034] running on GCE node
I20260810 14:57:50.986871 29637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:50.987108 29637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:50.988257 29637 hybrid_clock.cc:648] HybridClock initialized: now 1786373870988239 us; error 30 us; skew 500 ppm
Aug 10 14:57:50 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:50.990933 29637 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:57:50.991319 29637 webserver.cc:460] Webserver started at http://127.28.193.193:43237/ using document root <none> and password file <none>
I20260810 14:57:50.991508 29637 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:50.991547 29637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:50.991638 29637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:50.992631 29637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "3bbeed1bcea145329d827132888099b4"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "b75249e950fcfa81e0955a15bbcea27c"
server_key_iv: "c03dfd6c0ae84f1292415100143edf2d"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.992930 29637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "3bbeed1bcea145329d827132888099b4"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "b75249e950fcfa81e0955a15bbcea27c"
server_key_iv: "c03dfd6c0ae84f1292415100143edf2d"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.994251 29637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:57:50.994978 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:50.995119 29637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:50.995199 29637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "3bbeed1bcea145329d827132888099b4"
format_stamp: "Formatted at 2026-08-10 14:57:50 on dist-test-slave-cvwc"
server_key: "b75249e950fcfa81e0955a15bbcea27c"
server_key_iv: "c03dfd6c0ae84f1292415100143edf2d"
server_key_version: "encryptionkey@0"
I20260810 14:57:50.995265 29637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:51.018654 29637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:51.020807 29637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:51.020933 29637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:51.021158 29637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:51.021462 29637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:51.021507 29637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.021529 29637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:51.021540 29637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.027143 29637 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45045
I20260810 14:57:51.027187 29849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45045 every 8 connection(s)
I20260810 14:57:51.027511 29637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:57:51.030050 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29637
I20260810 14:57:51.030128 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:57:51.030277 29447 external_mini_cluster.cc:1428] Setting key 9d7863c37ad6d0abcabf703f91e48856
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373870, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:51.037671 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.028288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:46305 (local address 127.28.193.254:42059)
0810 14:57:51.028366 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:51.028368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:51.028502 (+ 134us) server_negotiation.cc:407] Connection header received
0810 14:57:51.029119 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:51.029120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:51.029135 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:51.029192 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:51.030392 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.031082 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.032109 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.032393 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.032459 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.034985 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:51.034987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:51.034987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:51.034995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:51.036389 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.036771 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.036772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.036773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.036826 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.037056 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.037057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.037058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.037166 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:51.037221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:51.037488 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 14:57:51.037581 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:57:51.038038 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.028216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42059 (local address 127.28.193.193:46305)
0810 14:57:51.028380 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:51.028424 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:51.029010 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:51.029266 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:51.029271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:51.030043 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:51.030290 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.030296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.031190 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.031191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.032018 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.032021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.032489 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.032491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.032660 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.033146 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:51.033156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:51.034891 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:51.036481 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.036488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.036491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.036703 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.036881 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.036881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.036882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.036999 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.037221 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:51.037227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:51.037366 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 14:57:51.037455 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:57:51.037610 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260810 14:57:51.038585 29851 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42059
I20260810 14:57:51.038674 29851 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:51.038846 29851 heartbeater.cc:511] Master 127.28.193.254:42059 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 14:57:51.039423 29536 ts_manager.cc:194] Registered new tserver with Master: 3bbeed1bcea145329d827132888099b4 (127.28.193.193:45045)
I20260810 14:57:51.040046 29536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:46305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:57:51.056483 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:51.138688 29858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:51.138883 29858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:51.138916 29858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:51.140015 29858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:51.140055 29858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:51.140089 29858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:57:51.141556 29858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:51.141824 29858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:51.142030 29858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:51.146883 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:51.146914 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:51.146982 29867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:51.147055 29858 server_base.cc:1034] running on GCE node
I20260810 14:57:51.147245 29858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:51.147460 29858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:51.148592 29858 hybrid_clock.cc:648] HybridClock initialized: now 1786373871148574 us; error 34 us; skew 500 ppm
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:51.151060 29858 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:57:51.151465 29858 webserver.cc:460] Webserver started at http://127.28.193.194:34787/ using document root <none> and password file <none>
I20260810 14:57:51.151667 29858 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:51.151707 29858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:51.151805 29858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:51.152783 29858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "c3223c9fcc804c84aa78b26a4aa593b0"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "80363b77f79f8522c08604b50c153c4d"
server_key_iv: "623be9856d26a4c2d694ea30332aa501"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.153071 29858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "c3223c9fcc804c84aa78b26a4aa593b0"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "80363b77f79f8522c08604b50c153c4d"
server_key_iv: "623be9856d26a4c2d694ea30332aa501"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.154349 29858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 14:57:51.155151 29882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.155324 29858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:51.155405 29858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "c3223c9fcc804c84aa78b26a4aa593b0"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "80363b77f79f8522c08604b50c153c4d"
server_key_iv: "623be9856d26a4c2d694ea30332aa501"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.155462 29858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:51.165531 29858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:51.167579 29858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:51.167692 29858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:51.167888 29858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:51.168148 29858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:51.168177 29858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.168203 29858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:51.168219 29858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.173975 29858 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:40939
I20260810 14:57:51.174012 30075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:40939 every 8 connection(s)
I20260810 14:57:51.174324 29858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:51.181452 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29858
I20260810 14:57:51.181543 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:57:51.181699 29447 external_mini_cluster.cc:1428] Setting key aa1c115dddb5af08eaac2e9f263f1667
I20260810 14:57:51.183259 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.175095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:50999 (local address 127.28.193.254:42059)
0810 14:57:51.175159 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:51.175161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:51.175378 (+ 217us) server_negotiation.cc:407] Connection header received
0810 14:57:51.175856 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:51.175858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:51.175869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:51.175920 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:51.176826 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.177322 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.178299 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.178510 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.178561 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.180570 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:51.180572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:51.180572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:51.180580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:51.181924 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.182326 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.182327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.182328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.182397 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.182610 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.182611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.182612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.182722 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:51.182776 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:51.183085 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 14:57:51.183172 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:57:51.183424 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.175069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42059 (local address 127.28.193.194:50999)
0810 14:57:51.175250 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:51.175301 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:51.175770 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:51.175979 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:51.175982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:51.176549 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:51.176742 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.176747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.177406 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.177406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.178229 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.178232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.178600 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.178601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.178718 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.179143 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:51.179151 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:51.180480 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:51.181989 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.181996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.181998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.182269 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.182451 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.182451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.182452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.182555 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.182774 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:51.182781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:51.182945 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 14:57:51.183053 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 14:57:51.183202 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260810 14:57:51.183846 30076 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42059
I20260810 14:57:51.183936 30076 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:51.184115 30076 heartbeater.cc:511] Master 127.28.193.254:42059 requested a full tablet report, sending...
I20260810 14:57:51.184484 29536 ts_manager.cc:194] Registered new tserver with Master: c3223c9fcc804c84aa78b26a4aa593b0 (127.28.193.194:40939)
I20260810 14:57:51.184832 29536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:50999
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:57:51.206519 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:51.288048 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:51.288249 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:51.288272 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:51.289376 30087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:51.289417 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:51.289440 30087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:57:51.290928 30087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:51.291108 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:51.291282 30087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:51.296232 30105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:51.296295 30087 server_base.cc:1034] running on GCE node
W20260810 14:57:51.296285 30101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:51.296233 30100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:51.296633 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:51.296866 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:51.298026 30087 hybrid_clock.cc:648] HybridClock initialized: now 1786373871298010 us; error 26 us; skew 500 ppm
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:51.300447 30087 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:57:51.300889 30087 webserver.cc:460] Webserver started at http://127.28.193.195:35779/ using document root <none> and password file <none>
I20260810 14:57:51.301082 30087 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:51.301129 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:51.301252 30087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:51.302289 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "f48da5689b224670afd8bbdc452bea5c"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "32875b631a8aa91fa2f8b6835f703719"
server_key_iv: "a3b86cc3b19e84854d5daf3d7b33fef1"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.302627 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "f48da5689b224670afd8bbdc452bea5c"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "32875b631a8aa91fa2f8b6835f703719"
server_key_iv: "a3b86cc3b19e84854d5daf3d7b33fef1"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.303910 30087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:51.304620 30117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.304785 30087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:51.304869 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "f48da5689b224670afd8bbdc452bea5c"
format_stamp: "Formatted at 2026-08-10 14:57:51 on dist-test-slave-cvwc"
server_key: "32875b631a8aa91fa2f8b6835f703719"
server_key_iv: "a3b86cc3b19e84854d5daf3d7b33fef1"
server_key_version: "encryptionkey@0"
I20260810 14:57:51.304927 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:51.311822 30087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:51.314462 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:51.314604 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:51.314846 30087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:51.315178 30087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:51.315213 30087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.315241 30087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:51.315259 30087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:51.321108 30087 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35179
I20260810 14:57:51.321201 30312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35179 every 8 connection(s)
I20260810 14:57:51.321467 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:51.331142 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30087
I20260810 14:57:51.331247 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:57:51.331452 29447 external_mini_cluster.cc:1428] Setting key 18ad714930a0833588d29ca9755a1d33
I20260810 14:57:51.332500 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.322339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:48861 (local address 127.28.193.254:42059)
0810 14:57:51.322393 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:51.322396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:51.322722 (+ 326us) server_negotiation.cc:407] Connection header received
0810 14:57:51.323406 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:51.323407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:51.323419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:51.323480 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:51.324726 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.325244 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.326876 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.327077 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.327142 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.329525 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:51.329527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:51.329528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:51.329536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:51.331002 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.331437 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.331446 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.331447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.331504 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.331830 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.331831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.331831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.331950 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:51.332007 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:51.332320 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 14:57:51.332400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 14:57:51.332742 30319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.322291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42059 (local address 127.28.193.195:48861)
0810 14:57:51.322570 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:51.322614 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:51.323263 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:51.323557 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:51.323562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:51.324300 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:51.324624 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.324631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.325340 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.325341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.326768 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:51.326772 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.327168 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.327169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:51.327337 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.327849 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:51.327859 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:51.329431 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:51.331094 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.331101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.331104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.331369 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.331559 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:51.331560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:51.331560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:51.331672 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:51.332012 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:51.332020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:51.332183 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 14:57:51.332299 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:57:51.332463 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260810 14:57:51.333336 30315 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42059
I20260810 14:57:51.333424 30315 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:51.333581 30315 heartbeater.cc:511] Master 127.28.193.254:42059 requested a full tablet report, sending...
I20260810 14:57:51.333932 29536 ts_manager.cc:194] Registered new tserver with Master: f48da5689b224670afd8bbdc452bea5c (127.28.193.195:35179)
I20260810 14:57:51.334457 29536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:48861
I20260810 14:57:51.343657 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:51.373255 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.364759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.28.193.193:45045)
0810 14:57:51.364943 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:51.364948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:51.364958 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:51.364982 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:51.364985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:51.364998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:51.365060 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:51.365633 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.366416 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.367127 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.367460 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.367522 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.370137 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:51.370147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:51.370147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:51.370156 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:51.371928 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.372256 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.372260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.372260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.372336 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.372564 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.372565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.372565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.372745 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:51.372909 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:51.373028 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:57:51.373120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
W20260810 14:57:51.373706 29703 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44210
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[29463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:51.400516 30329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:51.393841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.28.193.193:45045)
0810 14:57:51.393913 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:51.393915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:51.393958 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:57:51.394032 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:51.394033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:51.394047 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:51.394096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:51.394656 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.395144 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.395854 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:51.396062 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:51.396117 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:51.398425 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:51.398427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:51.398427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:51.398434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:51.399712 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.399997 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.399997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.399998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.400051 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:51.400172 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:51.400173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:51.400173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:51.400254 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:51.400338 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:51.400386 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 14:57:51.400451 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:57:51.401007 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29637
I20260810 14:57:51.415112 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29858
I20260810 14:57:51.428192 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30087
I20260810 14:57:51.441244 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29487
2026-08-10T14:57:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2100 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[30340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[30340](info): set up 2 sockets
Aug 10 14:57:51 dist-test-slave-cvwc krb5kdc[30340](info): commencing operation
krb5kdc: starting...
W20260810 14:57:52.637491 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:57:52 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:57:52.753224 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:43769
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:43769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:52.834875 30360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:52.835088 30360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:52.835117 30360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:52.836263 30360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:57:52.836304 30360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:52.836323 30360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:52.836338 30360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:57:52.836354 30360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:57:52.837852 30360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:43769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:43769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:52.838099 30360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:52.838317 30360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:52.843112 30372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:52.843123 30375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:52.843195 30360 server_base.cc:1034] running on GCE node
W20260810 14:57:52.843125 30371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:52.843542 30360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:52.843844 30360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:52.844992 30360 hybrid_clock.cc:648] HybridClock initialized: now 1786373872844977 us; error 29 us; skew 500 ppm
Aug 10 14:57:52 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373872, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:52.848220 30360 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:57:52.848737 30360 webserver.cc:460] Webserver started at http://127.28.193.254:45889/ using document root <none> and password file <none>
I20260810 14:57:52.849009 30360 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:52.849071 30360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:52.849206 30360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:52.850605 30360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "1fb25cf3727546988294ad6d5005fca3"
format_stamp: "Formatted at 2026-08-10 14:57:52 on dist-test-slave-cvwc"
server_key: "9a007904e061c9a6af786cf194a9ec82"
server_key_iv: "76da7fc05148c740a16ecd2f3a6fd065"
server_key_version: "encryptionkey@0"
I20260810 14:57:52.851020 30360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "1fb25cf3727546988294ad6d5005fca3"
format_stamp: "Formatted at 2026-08-10 14:57:52 on dist-test-slave-cvwc"
server_key: "9a007904e061c9a6af786cf194a9ec82"
server_key_iv: "76da7fc05148c740a16ecd2f3a6fd065"
server_key_version: "encryptionkey@0"
I20260810 14:57:52.852432 30360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:52.853217 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:52.853408 30360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:52.853518 30360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "1fb25cf3727546988294ad6d5005fca3"
format_stamp: "Formatted at 2026-08-10 14:57:52 on dist-test-slave-cvwc"
server_key: "9a007904e061c9a6af786cf194a9ec82"
server_key_iv: "76da7fc05148c740a16ecd2f3a6fd065"
server_key_version: "encryptionkey@0"
I20260810 14:57:52.853577 30360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:52.870733 30360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:52.872602 30360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:52.872716 30360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:52.876329 30360 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:43769
I20260810 14:57:52.876372 30477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:43769 every 8 connection(s)
I20260810 14:57:52.876694 30360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:57:52.877552 30478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:52.877972 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30360
I20260810 14:57:52.878049 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:57:52.878198 29447 external_mini_cluster.cc:1428] Setting key b02a532eca4be38c855246dbbe83c6a8
I20260810 14:57:52.880360 30478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: Bootstrap starting.
I20260810 14:57:52.880929 30478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:52.881199 30478 log.cc:824] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:52.881985 30478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: No bootstrap required, opened a new log
I20260810 14:57:52.884241 30478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } }
I20260810 14:57:52.884373 30478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:52.884402 30478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb25cf3727546988294ad6d5005fca3, State: Initialized, Role: FOLLOWER
Aug 10 14:57:52 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:52.884529 30478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } }
I20260810 14:57:52.884603 30478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:57:52.884629 30478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:57:52.884673 30478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:52.885272 30478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } }
I20260810 14:57:52.885363 30478 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb25cf3727546988294ad6d5005fca3; no voters:
I20260810 14:57:52.885532 30478 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:57:52.885581 30486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:52.885778 30486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [term 1 LEADER]: Becoming Leader. State: Replica: 1fb25cf3727546988294ad6d5005fca3, State: Running, Role: LEADER
I20260810 14:57:52.885828 30478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:57:52.885877 30486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } }
I20260810 14:57:52.886211 30486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb25cf3727546988294ad6d5005fca3. Latest consensus state: current_term: 1 leader_uuid: "1fb25cf3727546988294ad6d5005fca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } } }
I20260810 14:57:52.886293 30486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:52.887288 30489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb25cf3727546988294ad6d5005fca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb25cf3727546988294ad6d5005fca3" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 43769 } } }
W20260810 14:57:52.887697 30511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 14:57:52.887804 30489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3 [sys.catalog]: This master's current role is: LEADER
W20260810 14:57:52.887821 30511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:57:52.887996 30499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:57:52.888146 30499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:57:52.888865 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:52.878708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.28.193.254:43769)
0810 14:57:52.878869 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:52.878876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:52.878900 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:57:52.879305 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:52.879311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:52.879355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:52.879479 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:52.880201 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:52.881083 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:52.882006 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:52.882419 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:52.882508 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:52.884971 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:52.884976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:52.884976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:52.884988 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:52.887024 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:52.887435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:52.887437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:52.887438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:52.887542 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:52.887798 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:52.887799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:52.887799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:52.888172 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:52.888319 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:52.888583 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 14:57:52.888723 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260810 14:57:52.890069 30499 catalog_manager.cc:1347] Generated new cluster ID: aa4cc141c5f84590b416f1d743650f30
I20260810 14:57:52.890120 30499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:57:52.900173 30499 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:57:52.900640 30499 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:57:52.911561 30499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb25cf3727546988294ad6d5005fca3: Generated new TSK 0
I20260810 14:57:52.911737 30499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:57:52.941790 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:53.022431 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:53.022626 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:53.022658 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:53.023762 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:53.023805 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:53.023840 30520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:57:53.025359 30520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:53.025580 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:53.025808 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:53.030609 30531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.030691 30528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.030642 30527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:53.030833 30520 server_base.cc:1034] running on GCE node
I20260810 14:57:53.031023 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:53.031211 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:53.032341 30520 hybrid_clock.cc:648] HybridClock initialized: now 1786373873032323 us; error 34 us; skew 500 ppm
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:53.034778 30520 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:57:53.035183 30520 webserver.cc:460] Webserver started at http://127.28.193.193:34903/ using document root <none> and password file <none>
I20260810 14:57:53.035355 30520 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:53.035396 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:53.035486 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:53.036502 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "90ea78584f3b4d0586eb5275ae1e4e3e"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "2b2b92d9a5cb1ef43d442cb2ec74c9ec"
server_key_iv: "10f91fe0afa352d5582681f30c383c0a"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.036798 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "90ea78584f3b4d0586eb5275ae1e4e3e"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "2b2b92d9a5cb1ef43d442cb2ec74c9ec"
server_key_iv: "10f91fe0afa352d5582681f30c383c0a"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.037962 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:57:53.038664 30541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.038837 30520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:53.038928 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "90ea78584f3b4d0586eb5275ae1e4e3e"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "2b2b92d9a5cb1ef43d442cb2ec74c9ec"
server_key_iv: "10f91fe0afa352d5582681f30c383c0a"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.038988 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:53.048091 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:53.050076 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:53.050204 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:53.050402 30520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:53.050671 30520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:53.050704 30520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.050730 30520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:53.050755 30520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.056264 30520 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:37603
I20260810 14:57:53.056306 30724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:37603 every 8 connection(s)
I20260810 14:57:53.056607 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:53.065438 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.057273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:44789 (local address 127.28.193.254:43769)
0810 14:57:53.057330 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.057332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.057551 (+ 219us) server_negotiation.cc:407] Connection header received
0810 14:57:53.058618 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.058619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.058630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.058681 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.059229 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.059718 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.060647 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.060855 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.060911 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.062851 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.062852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.062853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.062860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.064140 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.064536 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.064537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.064537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.064593 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.064806 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.064807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.064807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.064920 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.064976 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.065231 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.065331 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:57:53.065557 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.057275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:43769 (local address 127.28.193.193:44789)
0810 14:57:53.057439 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.057474 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.058523 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.058737 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.058740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.058984 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:53.059168 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.059172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.059776 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.059777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.060565 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.060567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.060920 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.060921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.061015 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.061430 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:53.061439 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:53.062764 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:53.064229 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.064236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.064238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.064464 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.064648 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.064648 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.064649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.064748 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.064978 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:53.064985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:53.065114 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.065201 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.065350 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260810 14:57:53.066037 30728 heartbeater.cc:347] Connected to a master server at 127.28.193.254:43769
I20260810 14:57:53.066128 30728 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:53.066303 30728 heartbeater.cc:511] Master 127.28.193.254:43769 requested a full tablet report, sending...
I20260810 14:57:53.066744 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30520
I20260810 14:57:53.066825 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:57:53.066817 30413 ts_manager.cc:194] Registered new tserver with Master: 90ea78584f3b4d0586eb5275ae1e4e3e (127.28.193.193:37603)
I20260810 14:57:53.066995 29447 external_mini_cluster.cc:1428] Setting key 0101b8f38fe134de176e0698c65ee3c6
I20260810 14:57:53.067530 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:44789
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:57:53.090804 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:53.175565 30737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:53.175796 30737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:53.175822 30737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:53.176980 30737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:53.177021 30737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:53.177047 30737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:57:53.178642 30737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:53.178864 30737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:53.179077 30737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:53.183907 30747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.183948 30750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.184006 30743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:53.183940 30737 server_base.cc:1034] running on GCE node
I20260810 14:57:53.184211 30737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:53.184450 30737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:53.185590 30737 hybrid_clock.cc:648] HybridClock initialized: now 1786373873185574 us; error 28 us; skew 500 ppm
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:53.188087 30737 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:57:53.188491 30737 webserver.cc:460] Webserver started at http://127.28.193.194:44089/ using document root <none> and password file <none>
I20260810 14:57:53.188685 30737 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:53.188725 30737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:53.188817 30737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:53.189883 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "c234f6c886d74c1bae7ecaf097caaece"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "72c3008e62fda4e74548c4c76acfaf04"
server_key_iv: "3b6823cb3a60c43b7a8b5e95cdca0e15"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.190196 30737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "c234f6c886d74c1bae7ecaf097caaece"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "72c3008e62fda4e74548c4c76acfaf04"
server_key_iv: "3b6823cb3a60c43b7a8b5e95cdca0e15"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.191463 30737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:53.192162 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.192355 30737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:57:53.192449 30737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "c234f6c886d74c1bae7ecaf097caaece"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "72c3008e62fda4e74548c4c76acfaf04"
server_key_iv: "3b6823cb3a60c43b7a8b5e95cdca0e15"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.192514 30737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:53.200727 30737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:53.202790 30737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:53.202934 30737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:53.203145 30737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:53.203444 30737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:53.203478 30737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.203500 30737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:53.203523 30737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.208849 30737 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:39307
I20260810 14:57:53.208909 30935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:39307 every 8 connection(s)
I20260810 14:57:53.209203 30737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:53.215886 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30737
I20260810 14:57:53.215967 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:57:53.216142 29447 external_mini_cluster.cc:1428] Setting key 58e92aa448d78ecd6f62eeed40e5852e
I20260810 14:57:53.218501 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.209952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:41523 (local address 127.28.193.254:43769)
0810 14:57:53.210015 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.210017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.210212 (+ 195us) server_negotiation.cc:407] Connection header received
0810 14:57:53.211183 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.211185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.211198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.211248 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.211874 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.212390 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.213375 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.213590 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.213647 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.215742 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.215743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.215744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.215752 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.217162 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.217535 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.217536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.217536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.217589 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.217856 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.217857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.217858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.217968 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.218022 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.218338 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.218424 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:57:53.218688 30938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.209932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:43769 (local address 127.28.193.194:41523)
0810 14:57:53.210094 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.210131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.211069 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.211323 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.211328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.211580 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:53.211767 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.211771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.212477 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.212478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.213285 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.213288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.213661 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.213662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.213837 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.214324 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:53.214332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:53.215641 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:53.217241 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.217247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.217250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.217468 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.217643 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.217644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.217644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.217783 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.218022 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:53.218029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:53.218162 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.218281 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.218448 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260810 14:57:53.219132 30936 heartbeater.cc:347] Connected to a master server at 127.28.193.254:43769
I20260810 14:57:53.219236 30936 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:53.219434 30936 heartbeater.cc:511] Master 127.28.193.254:43769 requested a full tablet report, sending...
I20260810 14:57:53.219844 30413 ts_manager.cc:194] Registered new tserver with Master: c234f6c886d74c1bae7ecaf097caaece (127.28.193.194:39307)
I20260810 14:57:53.220208 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:41523
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:57:53.240610 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:53.320943 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:53.321146 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:53.321177 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:53.322352 30943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:53.322395 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:53.322429 30943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:57:53.323873 30943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:43769
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:53.324097 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:53.324316 30943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:53.329082 30954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.329103 30950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:53.329078 30951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:53.329169 30943 server_base.cc:1034] running on GCE node
I20260810 14:57:53.329483 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:53.329744 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:53.330900 30943 hybrid_clock.cc:648] HybridClock initialized: now 1786373873330882 us; error 35 us; skew 500 ppm
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:53.333489 30943 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:57:53.333927 30943 webserver.cc:460] Webserver started at http://127.28.193.195:38037/ using document root <none> and password file <none>
I20260810 14:57:53.334122 30943 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:53.334168 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:53.334297 30943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:53.335335 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "e6b6394c81b545d9b4de62156bd99fc3"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "fdf0efc38579dbbd0ddf9e117e93e525"
server_key_iv: "6825b41e656bf1a902b5cbae5f08cce2"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.335662 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "e6b6394c81b545d9b4de62156bd99fc3"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "fdf0efc38579dbbd0ddf9e117e93e525"
server_key_iv: "6825b41e656bf1a902b5cbae5f08cce2"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.336892 30943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:57:53.337589 30963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.337766 30943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:57:53.337853 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "e6b6394c81b545d9b4de62156bd99fc3"
format_stamp: "Formatted at 2026-08-10 14:57:53 on dist-test-slave-cvwc"
server_key: "fdf0efc38579dbbd0ddf9e117e93e525"
server_key_iv: "6825b41e656bf1a902b5cbae5f08cce2"
server_key_version: "encryptionkey@0"
I20260810 14:57:53.337919 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:53.350358 30943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:53.351939 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:53.352072 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:53.352257 30943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:57:53.352538 30943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:53.352571 30943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.352597 30943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:53.352614 30943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:53.358021 30943 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:39317
I20260810 14:57:53.358083 31144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:39317 every 8 connection(s)
I20260810 14:57:53.358362 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:53.365669 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30943
I20260810 14:57:53.365778 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:57:53.365942 29447 external_mini_cluster.cc:1428] Setting key d7dac5e9af53f19727f5b43b54b9cf0f
I20260810 14:57:53.366878 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.359050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:59813 (local address 127.28.193.254:43769)
0810 14:57:53.359109 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.359111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.359274 (+ 163us) server_negotiation.cc:407] Connection header received
0810 14:57:53.360179 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.360181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.360194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.360245 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.360834 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.361329 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.362300 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.362498 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.362552 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.364471 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.364473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.364473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.364480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.365573 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.366004 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.366005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.366006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.366059 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.366295 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.366296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.366296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.366416 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.366472 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.366707 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.366784 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:57:53.367069 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.359013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:43769 (local address 127.28.193.195:59813)
0810 14:57:53.359168 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.359202 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.360076 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.360312 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.360316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.360567 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:53.360746 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.360750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.361409 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.361410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.362219 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.362222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.362565 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.362565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.362675 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.363116 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:53.363124 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:53.364387 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:53.365646 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.365652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.365654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.365938 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.366139 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:53.366141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:53.366141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:53.366247 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:53.366471 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:53.366477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:53.366605 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.366685 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.366842 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260810 14:57:53.367452 31145 heartbeater.cc:347] Connected to a master server at 127.28.193.254:43769
I20260810 14:57:53.367527 31145 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:53.367686 31145 heartbeater.cc:511] Master 127.28.193.254:43769 requested a full tablet report, sending...
I20260810 14:57:53.368019 30413 ts_manager.cc:194] Registered new tserver with Master: e6b6394c81b545d9b4de62156bd99fc3 (127.28.193.195:39317)
I20260810 14:57:53.368362 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:59813
I20260810 14:57:53.377542 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:57:53.384814 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.379236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.28.193.254:43769)
0810 14:57:53.379296 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.379298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.379373 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:57:53.379436 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.379436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.379447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.379493 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.380042 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.380552 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.381234 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.381424 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.381479 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.382599 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.382600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.382601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.382608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.383788 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.384090 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.384091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.384091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.384147 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.384301 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.384302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.384302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.384458 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.384548 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.384652 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.384725 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:57:53.386914 30413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:57:53.388480 30413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:53.394869 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.390623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:37603 (local address 127.0.0.1:36466)
0810 14:57:53.390938 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.390981 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.391288 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.391504 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.391505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.391508 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:53.391571 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.391573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.392486 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.392488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.393665 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.393668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.394567 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.394567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.394652 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.394669 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.394733 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.394787 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":60,"threads_started":1}
I20260810 14:57:53.394907 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.390771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.28.193.193:37603)
0810 14:57:53.390917 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.390922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.391254 (+ 332us) server_negotiation.cc:407] Connection header received
0810 14:57:53.391405 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.391408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.391424 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.391662 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:53.391749 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.392392 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.393790 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.394481 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.394560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.394618 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.394753 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.394795 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260810 14:57:53.395107 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.390792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:39307 (local address 127.0.0.1:57998)
0810 14:57:53.390965 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.390982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.391053 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.391398 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.391401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.391403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:53.391543 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.391546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.392825 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.392826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.393871 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.393873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.394685 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.394686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.394749 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.394765 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.394818 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.394944 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":1,"thread_start_us":46,"threads_started":1}
I20260810 14:57:53.395198 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.390938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.28.193.194:39307)
0810 14:57:53.391242 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.391246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.391253 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:57:53.391284 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.391285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.391297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.391391 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:53.391626 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.392720 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.393938 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.394621 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.394685 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.394732 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.394832 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.394873 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
I20260810 14:57:53.395776 30615 tablet_service.cc:1467] Processing CreateTablet for tablet 9787db42d36c4af6a725f4a15098eb05 (DEFAULT_TABLE table=test-table [id=0e6b803f84c94401bb9317f22d93df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:53.395825 30831 tablet_service.cc:1467] Processing CreateTablet for tablet 9787db42d36c4af6a725f4a15098eb05 (DEFAULT_TABLE table=test-table [id=0e6b803f84c94401bb9317f22d93df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:53.396162 30615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9787db42d36c4af6a725f4a15098eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:53.396174 30831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9787db42d36c4af6a725f4a15098eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:53.396341 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.390919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39317 (local address 127.0.0.1:39602)
0810 14:57:53.391101 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.391111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.391155 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.391490 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.391491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.391493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:53.391543 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.391546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.393935 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.393936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.394612 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.394614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.396138 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.396139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.396197 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.396201 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.396244 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.396280 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":5,"thread_start_us":39,"threads_started":1}
I20260810 14:57:53.396622 31164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.391089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.28.193.195:39317)
0810 14:57:53.391270 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.391274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.391283 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:57:53.391320 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.391323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.391339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.391408 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:53.391625 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.392720 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.395190 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.395967 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.396037 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.396485 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.396513 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.396550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260810 14:57:53.397203 31045 tablet_service.cc:1467] Processing CreateTablet for tablet 9787db42d36c4af6a725f4a15098eb05 (DEFAULT_TABLE table=test-table [id=0e6b803f84c94401bb9317f22d93df3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:53.397523 31045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9787db42d36c4af6a725f4a15098eb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:53.398937 31166 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Bootstrap starting.
I20260810 14:57:53.399431 31167 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Bootstrap starting.
I20260810 14:57:53.399682 31166 tablet_bootstrap.cc:654] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:53.400074 31167 tablet_bootstrap.cc:654] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:53.400326 31166 log.cc:824] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:53.400552 31168 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Bootstrap starting.
I20260810 14:57:53.400622 31167 log.cc:824] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:53.401167 31168 tablet_bootstrap.cc:654] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:53.401343 31166 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: No bootstrap required, opened a new log
I20260810 14:57:53.401353 31167 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: No bootstrap required, opened a new log
I20260810 14:57:53.401418 31167 ts_tablet_manager.cc:1397] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:53.401418 31166 ts_tablet_manager.cc:1397] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:57:53.401659 31168 log.cc:824] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:53.402477 31168 tablet_bootstrap.cc:492] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: No bootstrap required, opened a new log
I20260810 14:57:53.402546 31168 ts_tablet_manager.cc:1397] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:53.403358 31167 raft_consensus.cc:348] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.403349 31166 raft_consensus.cc:348] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.403537 31166 raft_consensus.cc:374] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:53.403537 31167 raft_consensus.cc:374] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:53.403573 31166 raft_consensus.cc:729] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ea78584f3b4d0586eb5275ae1e4e3e, State: Initialized, Role: FOLLOWER
I20260810 14:57:53.403573 31167 raft_consensus.cc:729] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c234f6c886d74c1bae7ecaf097caaece, State: Initialized, Role: FOLLOWER
I20260810 14:57:53.403687 31166 consensus_queue.cc:260] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.403702 31167 consensus_queue.cc:260] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.403981 31166 ts_tablet_manager.cc:1428] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:57:53.403983 30936 heartbeater.cc:503] Master 127.28.193.254:43769 was elected leader, sending a full tablet report...
I20260810 14:57:53.403981 31167 ts_tablet_manager.cc:1428] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:57:53.403947 31168 raft_consensus.cc:348] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.404099 30728 heartbeater.cc:503] Master 127.28.193.254:43769 was elected leader, sending a full tablet report...
I20260810 14:57:53.404106 31168 raft_consensus.cc:374] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:53.404139 31168 raft_consensus.cc:729] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b6394c81b545d9b4de62156bd99fc3, State: Initialized, Role: FOLLOWER
I20260810 14:57:53.404242 31168 consensus_queue.cc:260] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.404520 31168 ts_tablet_manager.cc:1428] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:53.404654 31145 heartbeater.cc:503] Master 127.28.193.254:43769 was elected leader, sending a full tablet report...
W20260810 14:57:53.459769 30937 tablet.cc:2369] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:57:53.515866 31179 raft_consensus.cc:482] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:53.515996 31179 raft_consensus.cc:504] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.516309 31179 leader_election.cc:288] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90ea78584f3b4d0586eb5275ae1e4e3e (127.28.193.193:37603), c234f6c886d74c1bae7ecaf097caaece (127.28.193.194:39307)
I20260810 14:57:53.520257 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.516657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44931 (local address 127.28.193.194:39307)
0810 14:57:53.516720 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.516721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.516849 (+ 128us) server_negotiation.cc:407] Connection header received
0810 14:57:53.516915 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.516916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.516924 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.516961 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:53.517091 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.517799 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519290 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.519867 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519942 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.519951 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.520151 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.520187 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:57:53.520257 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.516585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:60185 (local address 127.28.193.193:37603)
0810 14:57:53.516639 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.516641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.516650 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:57:53.516681 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.516682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.516692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.516788 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:53.517006 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.517787 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519293 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.519866 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519942 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.519951 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.520151 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.520188 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 14:57:53.520280 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.516481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:37603 (local address 127.28.193.195:60185)
0810 14:57:53.516537 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.516549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.516593 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.516848 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.516849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.516851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:53.516922 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.516924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.517877 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.517878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.519201 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.519205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.519972 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.520078 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.520080 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.520140 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.520188 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":143}
I20260810 14:57:53.520298 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.516581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:39307 (local address 127.28.193.195:44931)
0810 14:57:53.516777 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:53.516788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:53.516853 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:53.516976 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:53.516977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:53.516979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:53.517035 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.517036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.517872 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.517873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.519201 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:53.519205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.519972 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.519973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:53.520069 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.520075 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:57:53.520140 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 14:57:53.520187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260810 14:57:53.520684 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9787db42d36c4af6a725f4a15098eb05" candidate_uuid: "e6b6394c81b545d9b4de62156bd99fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c234f6c886d74c1bae7ecaf097caaece" is_pre_election: true
I20260810 14:57:53.520694 30651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9787db42d36c4af6a725f4a15098eb05" candidate_uuid: "e6b6394c81b545d9b4de62156bd99fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" is_pre_election: true
I20260810 14:57:53.520834 30651 raft_consensus.cc:2454] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b6394c81b545d9b4de62156bd99fc3 in term 0.
I20260810 14:57:53.520838 30864 raft_consensus.cc:2454] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6b6394c81b545d9b4de62156bd99fc3 in term 0.
I20260810 14:57:53.521071 30970 leader_election.cc:302] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ea78584f3b4d0586eb5275ae1e4e3e, e6b6394c81b545d9b4de62156bd99fc3; no voters:
I20260810 14:57:53.521222 31179 raft_consensus.cc:2781] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:53.521291 31179 raft_consensus.cc:482] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:53.521313 31179 raft_consensus.cc:3037] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:53.522069 31179 raft_consensus.cc:504] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.522186 31179 leader_election.cc:288] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 90ea78584f3b4d0586eb5275ae1e4e3e (127.28.193.193:37603), c234f6c886d74c1bae7ecaf097caaece (127.28.193.194:39307)
I20260810 14:57:53.522439 30651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9787db42d36c4af6a725f4a15098eb05" candidate_uuid: "e6b6394c81b545d9b4de62156bd99fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e"
I20260810 14:57:53.522476 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9787db42d36c4af6a725f4a15098eb05" candidate_uuid: "e6b6394c81b545d9b4de62156bd99fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c234f6c886d74c1bae7ecaf097caaece"
I20260810 14:57:53.522514 30651 raft_consensus.cc:3037] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:53.522557 30864 raft_consensus.cc:3037] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:53.523185 30651 raft_consensus.cc:2454] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b6394c81b545d9b4de62156bd99fc3 in term 1.
I20260810 14:57:53.523257 30864 raft_consensus.cc:2454] T 9787db42d36c4af6a725f4a15098eb05 P c234f6c886d74c1bae7ecaf097caaece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6b6394c81b545d9b4de62156bd99fc3 in term 1.
I20260810 14:57:53.523350 30970 leader_election.cc:302] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90ea78584f3b4d0586eb5275ae1e4e3e, e6b6394c81b545d9b4de62156bd99fc3; no voters:
I20260810 14:57:53.523442 31179 raft_consensus.cc:2781] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:53.523600 31179 raft_consensus.cc:686] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [term 1 LEADER]: Becoming Leader. State: Replica: e6b6394c81b545d9b4de62156bd99fc3, State: Running, Role: LEADER
I20260810 14:57:53.523696 31179 consensus_queue.cc:237] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } }
I20260810 14:57:53.524576 30409 catalog_manager.cc:5495] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6b6394c81b545d9b4de62156bd99fc3 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "e6b6394c81b545d9b4de62156bd99fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ea78584f3b4d0586eb5275ae1e4e3e" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c234f6c886d74c1bae7ecaf097caaece" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6b6394c81b545d9b4de62156bd99fc3" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39317 } health_report { overall_health: HEALTHY } } }
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:53.546241 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.538964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.28.193.193:37603)
0810 14:57:53.539044 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.539046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.539114 (+ 68us) server_negotiation.cc:407] Connection header received
0810 14:57:53.539205 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.539206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.539226 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.539275 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.539867 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.540377 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.541115 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.541308 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.541369 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.543753 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.543763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.543763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.543770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.545248 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.545572 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.545575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.545576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.545629 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.545823 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.545824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.545824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.545953 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.546059 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.546118 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.546178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:57:53.557337 30729 tablet.cc:2369] T 9787db42d36c4af6a725f4a15098eb05 P 90ea78584f3b4d0586eb5275ae1e4e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Password for test-user@KRBTEST.COM:
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:53.569171 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.562633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.28.193.193:37603)
0810 14:57:53.562712 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.562714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.562724 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:53.562807 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.562808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.562818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.562866 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.563383 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.563851 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.564527 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.564713 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.564769 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.566946 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.566947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.566948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.566954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.568064 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.568414 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.568415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.568416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.568472 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.568679 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.568679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.568680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.568789 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.568864 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.569021 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.569085 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[30340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:53.590421 31159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:53.584277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.28.193.193:37603)
0810 14:57:53.584355 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:53.584357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:53.584367 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:53.584413 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:53.584414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:53.584423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:53.584470 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:53.584950 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.585400 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.586161 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:53.586345 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:53.586426 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:53.588423 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:53.588424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:53.588425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:53.588432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:53.589447 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.589785 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.589786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.589786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.589833 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:53.589997 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:53.589998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:53.589999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:53.590086 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:53.590166 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:53.590263 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:57:53.590332 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:57:53.590870 30584 tablet_service.cc:2908] Rejecting scan request for tablet 9787db42d36c4af6a725f4a15098eb05: Uninitialized: clean time has not yet been initialized
I20260810 14:57:53.591728 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30520
I20260810 14:57:53.602180 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30737
W20260810 14:57:53.608909 31146 tablet.cc:2369] T 9787db42d36c4af6a725f4a15098eb05 P e6b6394c81b545d9b4de62156bd99fc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:57:53.612221 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30943
I20260810 14:57:53.635051 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30360
2026-08-10T14:57:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2193 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[31210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[31210](info): set up 2 sockets
Aug 10 14:57:53 dist-test-slave-cvwc krb5kdc[31210](info): commencing operation
krb5kdc: starting...
W20260810 14:57:54.796794 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:57:54 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:57:54.923882 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34845
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:55.005754 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:55.005985 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:55.006013 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:55.007128 31226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:57:55.007169 31226 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:57:55.007185 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:55.007205 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:55.007220 31226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:57:55.007236 31226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:57:55.008694 31226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:55.008958 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:55.009169 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:55.013914 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.013914 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.013911 31234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:55.014010 31226 server_base.cc:1034] running on GCE node
I20260810 14:57:55.014364 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:55.014611 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:55.015741 31226 hybrid_clock.cc:648] HybridClock initialized: now 1786373875015724 us; error 30 us; skew 500 ppm
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:55.018128 31226 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:57:55.018548 31226 webserver.cc:460] Webserver started at http://127.28.193.254:41335/ using document root <none> and password file <none>
I20260810 14:57:55.018745 31226 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:55.018784 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:55.018903 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:55.019886 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "38bd5582b9bc4392bdf98b28650cab57"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "fcd46c8099d40ae9fd66601b6564464f"
server_key_iv: "8e28b7f91619abf7e75a11a376615c34"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.020193 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "38bd5582b9bc4392bdf98b28650cab57"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "fcd46c8099d40ae9fd66601b6564464f"
server_key_iv: "8e28b7f91619abf7e75a11a376615c34"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.021410 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:55.022140 31254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.022266 31226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.022337 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "38bd5582b9bc4392bdf98b28650cab57"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "fcd46c8099d40ae9fd66601b6564464f"
server_key_iv: "8e28b7f91619abf7e75a11a376615c34"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.022404 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:55.035558 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:55.037181 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:55.037312 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:55.041978 31226 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34845
I20260810 14:57:55.042044 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34845 every 8 connection(s)
I20260810 14:57:55.042344 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:57:55.043010 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.045199 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Bootstrap starting.
I20260810 14:57:55.045778 31345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.046029 31345 log.cc:824] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:55.046691 31345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: No bootstrap required, opened a new log
I20260810 14:57:55.048920 31345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } }
I20260810 14:57:55.049060 31345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.049081 31345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bd5582b9bc4392bdf98b28650cab57, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.049208 31345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } }
I20260810 14:57:55.049285 31345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:57:55.049316 31345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:57:55.049364 31345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.049408 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31226
I20260810 14:57:55.049504 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:57:55.049659 29447 external_mini_cluster.cc:1428] Setting key d6fe46aab3fe20c3d74c4a314f4e6c65
I20260810 14:57:55.049960 31345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } }
I20260810 14:57:55.050060 31345 leader_election.cc:302] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38bd5582b9bc4392bdf98b28650cab57; no voters:
I20260810 14:57:55.050223 31345 leader_election.cc:288] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:57:55.050257 31350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:55.050395 31350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [term 1 LEADER]: Becoming Leader. State: Replica: 38bd5582b9bc4392bdf98b28650cab57, State: Running, Role: LEADER
I20260810 14:57:55.050498 31345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:57:55.050479 31350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } }
I20260810 14:57:55.051021 31353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38bd5582b9bc4392bdf98b28650cab57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } } }
I20260810 14:57:55.051021 31354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38bd5582b9bc4392bdf98b28650cab57. Latest consensus state: current_term: 1 leader_uuid: "38bd5582b9bc4392bdf98b28650cab57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38bd5582b9bc4392bdf98b28650cab57" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34845 } } }
I20260810 14:57:55.051141 31353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:55.051146 31354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57 [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:55.051894 31365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:57:55.052423 31365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:57:55.054028 31365 catalog_manager.cc:1347] Generated new cluster ID: 425d7e1a4e2f4bba9079f5428ce7fb31
I20260810 14:57:55.054078 31365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:55.060283 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.050093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.28.193.254:34845)
0810 14:57:55.050243 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.050253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.050288 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:57:55.050599 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.050608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.050669 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.050825 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.052119 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.053137 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.054062 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.054475 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.054570 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.056704 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.056708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.056708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.056720 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.058914 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.059241 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.059244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.059244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.059338 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.059519 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.059519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.059520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.059745 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.059882 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.059984 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.060117 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260810 14:57:55.080969 31365 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:57:55.081753 31365 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:57:55.093042 31365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Generated new TSK 0
I20260810 14:57:55.093235 31365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:57:55.122328 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34845
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:55.203346 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:55.203549 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:55.203574 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:55.204682 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:55.204721 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:55.204741 31387 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:55.204770 31387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:57:55.206207 31387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:55.206427 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:55.206638 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:55.211484 31395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.211463 31398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:55.211578 31387 server_base.cc:1034] running on GCE node
W20260810 14:57:55.211463 31394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:55.211869 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:55.212082 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:55.213233 31387 hybrid_clock.cc:648] HybridClock initialized: now 1786373875213220 us; error 29 us; skew 500 ppm
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:55.215673 31387 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:57:55.216068 31387 webserver.cc:460] Webserver started at http://127.28.193.193:35721/ using document root <none> and password file <none>
I20260810 14:57:55.216264 31387 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:55.216311 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:55.216442 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:55.217447 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "37f58309c02741349922934827bea433"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "9de00f33e399966b1114868392a6b171"
server_key_iv: "dcd4d251f5c83932cbe299f068fa290f"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.217821 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "37f58309c02741349922934827bea433"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "9de00f33e399966b1114868392a6b171"
server_key_iv: "dcd4d251f5c83932cbe299f068fa290f"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.219062 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:55.219720 31410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.219862 31387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:55.219946 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "37f58309c02741349922934827bea433"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "9de00f33e399966b1114868392a6b171"
server_key_iv: "dcd4d251f5c83932cbe299f068fa290f"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.220013 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:55.235126 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:55.236963 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:55.237097 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:55.237594 31387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:55.237632 31387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.237659 31387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:55.237677 31387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.244462 31387 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45875
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:55.244601 31583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45875 every 8 connection(s)
I20260810 14:57:55.244828 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:57:55.247447 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.238215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:41921 (local address 127.28.193.254:34845)
0810 14:57:55.238277 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.238280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.238453 (+ 173us) server_negotiation.cc:407] Connection header received
0810 14:57:55.239131 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.239132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.239143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.239203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.240059 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.240829 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.242312 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.242579 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.242634 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.245126 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.245128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.245128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.245136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.246446 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.246781 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.246782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.246782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.246826 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.247001 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.247002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.247002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.247096 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.247141 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.247325 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.247384 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:57:55.247598 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.238175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.193:41921)
0810 14:57:55.238320 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.238368 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.239028 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.239297 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.239302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.239655 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.239953 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.239961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.240920 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.240929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.242227 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.242231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.242658 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.242659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.242798 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.243284 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.243294 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.245018 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.246523 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.246528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.246530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.246727 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.246867 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.246868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.246868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.246960 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.247137 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.247139 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.247240 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.247313 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.247422 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260810 14:57:55.247859 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31387
I20260810 14:57:55.247977 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:57:55.248152 29447 external_mini_cluster.cc:1428] Setting key b7ca2519c9b3bc413b3eaca9b88c9b5b
I20260810 14:57:55.248994 31584 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34845
I20260810 14:57:55.249078 31584 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:55.249245 31584 heartbeater.cc:511] Master 127.28.193.254:34845 requested a full tablet report, sending...
I20260810 14:57:55.249704 31278 ts_manager.cc:194] Registered new tserver with Master: 37f58309c02741349922934827bea433 (127.28.193.193:45875)
I20260810 14:57:55.250322 31278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:41921
I20260810 14:57:55.254927 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.249260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:43883 (local address 127.28.193.254:34845)
0810 14:57:55.249297 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.249298 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.249307 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:57:55.249396 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.249396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.249406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.249463 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.249976 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.250642 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.251529 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.251726 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.251778 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.252626 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.252627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.252627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.252634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.253983 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.254269 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.254270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.254270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.254321 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.254476 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.254477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.254478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.254576 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.254629 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.254770 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.254837 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 14:57:55.254951 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.249092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.193:43883)
0810 14:57:55.249222 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.249234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.249342 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.249533 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.249534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.249706 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.249892 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.249895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.250801 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.250802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.251460 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.251463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.251782 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.251783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.251855 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.252260 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.252264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.252552 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.254060 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.254061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.254061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.254213 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.254371 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.254371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.254372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.254427 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.254627 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.254628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.254684 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.254741 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.254837 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":51,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:57:55.274497 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34845
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:55.356354 31597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:55.356560 31597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:55.356583 31597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:55.357684 31597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:55.357759 31597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:55.357786 31597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:55.357802 31597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:57:55.359315 31597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:55.359557 31597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:55.359766 31597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:55.364581 31608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.364589 31606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.364589 31605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:55.364827 31597 server_base.cc:1034] running on GCE node
I20260810 14:57:55.365039 31597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:55.365260 31597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:55.366437 31597 hybrid_clock.cc:648] HybridClock initialized: now 1786373875366422 us; error 30 us; skew 500 ppm
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:55.368970 31597 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:57:55.369378 31597 webserver.cc:460] Webserver started at http://127.28.193.194:45395/ using document root <none> and password file <none>
I20260810 14:57:55.369606 31597 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:55.369654 31597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:55.369803 31597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:55.370862 31597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "aa7f01d6db724975adf662b0f713eb2a"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "dd5104aab0e016e293fb7ee807c3f9b8"
server_key_iv: "cd7576535b7142cbea2f25fddfc5fb4b"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.371191 31597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "aa7f01d6db724975adf662b0f713eb2a"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "dd5104aab0e016e293fb7ee807c3f9b8"
server_key_iv: "cd7576535b7142cbea2f25fddfc5fb4b"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.372437 31597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:55.373159 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.373327 31597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:55.373409 31597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "aa7f01d6db724975adf662b0f713eb2a"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "dd5104aab0e016e293fb7ee807c3f9b8"
server_key_iv: "cd7576535b7142cbea2f25fddfc5fb4b"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.373471 31597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:55.387012 31597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:55.388685 31597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:55.388820 31597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:55.389321 31597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:55.389359 31597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.389387 31597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:55.389407 31597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:55.396041 31597 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:42069
I20260810 14:57:55.396199 31798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:42069 every 8 connection(s)
I20260810 14:57:55.396418 31597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:57:55.398751 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.389762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:46523 (local address 127.28.193.254:34845)
0810 14:57:55.389822 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.389825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.390050 (+ 225us) server_negotiation.cc:407] Connection header received
0810 14:57:55.390693 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.390694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.390706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.390750 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.391576 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.392323 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.393812 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.394064 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.394124 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.396574 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.396576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.396577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.396586 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.397695 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.398060 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.398060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.398061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.398107 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.398265 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.398265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.398266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.398359 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.398403 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.398628 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.398689 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:57:55.398900 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.389706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.194:46523)
0810 14:57:55.389912 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.389959 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.390592 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.390826 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.390830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.391180 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.391472 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.391479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.392430 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.392431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.393699 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.393702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.394143 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.394144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.394286 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.394736 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.394747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.396485 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.397779 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.397786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.397789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.397998 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.398142 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.398143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.398143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.398226 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.398409 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.398412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.398529 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.398608 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.398724 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260810 14:57:55.399333 31801 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34845
I20260810 14:57:55.399413 31801 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:55.399504 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31597
I20260810 14:57:55.399592 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:57:55.399595 31801 heartbeater.cc:511] Master 127.28.193.254:34845 requested a full tablet report, sending...
I20260810 14:57:55.399776 29447 external_mini_cluster.cc:1428] Setting key f77b2e809aca3cc8b9d154c22de9d392
I20260810 14:57:55.399971 31273 ts_manager.cc:194] Registered new tserver with Master: aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069)
I20260810 14:57:55.400406 31273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:46523
I20260810 14:57:55.407174 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.400355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.194:59695)
0810 14:57:55.401825 (+ 1470us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.401838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.401904 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.402029 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.402030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.402182 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.402246 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.402247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.403058 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.403058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.403563 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.403564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.403961 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.403962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.404009 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.404418 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.404421 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.404741 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.406271 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.406272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.406273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.406437 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.406615 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.406615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.406615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.406660 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.406948 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.406949 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.406996 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.407040 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.407106 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"thread_start_us":61,"threads_started":1}
I20260810 14:57:55.407230 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.400504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:59695 (local address 127.28.193.254:34845)
0810 14:57:55.400585 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.400587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.401904 (+ 1317us) server_negotiation.cc:407] Connection header received
0810 14:57:55.401955 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.401956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.401968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.402025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.402303 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.402961 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.403633 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.403881 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.403965 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.404826 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.404828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.404828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.404835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.406197 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.406503 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.406504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.406504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.406560 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.406696 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.406697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.406697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.406876 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.406961 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.407066 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.407142 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:57:55.426512 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34845
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:55.508023 31816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:55.508211 31816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:55.508234 31816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:55.509356 31816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:55.509393 31816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:55.509411 31816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:55.509428 31816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:57:55.510829 31816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:55.511042 31816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:55.511247 31816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:55.516011 31821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.516086 31822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:55.516036 31824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:55.516216 31816 server_base.cc:1034] running on GCE node
I20260810 14:57:55.516419 31816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:55.516606 31816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:55.517764 31816 hybrid_clock.cc:648] HybridClock initialized: now 1786373875517747 us; error 29 us; skew 500 ppm
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:55.520144 31816 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:57:55.520545 31816 webserver.cc:460] Webserver started at http://127.28.193.195:43579/ using document root <none> and password file <none>
I20260810 14:57:55.520721 31816 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:55.520759 31816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:55.520850 31816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:55.521854 31816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "257f8081402546d5a98c3fa89f6b2cca"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "a03ca6a3487378200129dd7a414a00e6"
server_key_iv: "88d344fdefdf99426e90339798249306"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.522195 31816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "257f8081402546d5a98c3fa89f6b2cca"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "a03ca6a3487378200129dd7a414a00e6"
server_key_iv: "88d344fdefdf99426e90339798249306"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.523484 31816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:55.524106 31835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.524278 31816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:57:55.524354 31816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "257f8081402546d5a98c3fa89f6b2cca"
format_stamp: "Formatted at 2026-08-10 14:57:55 on dist-test-slave-cvwc"
server_key: "a03ca6a3487378200129dd7a414a00e6"
server_key_iv: "88d344fdefdf99426e90339798249306"
server_key_version: "encryptionkey@0"
I20260810 14:57:55.524423 31816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:55.543624 31816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:55.545756 31816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:55.545889 31816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:55.546407 31816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:55.546466 31816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.546497 31816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:55.546514 31816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:55.553203 31816 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:36075
I20260810 14:57:55.553270 32029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:36075 every 8 connection(s)
I20260810 14:57:55.553583 31816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:57:55.556175 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.546809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40387 (local address 127.28.193.254:34845)
0810 14:57:55.546880 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.546882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.547077 (+ 195us) server_negotiation.cc:407] Connection header received
0810 14:57:55.547572 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.547573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.547587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.547643 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.548568 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.549310 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.550811 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.551099 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.551163 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.553611 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.553613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.553613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.553621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.554963 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.555315 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.555316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.555317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.555374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.555584 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.555585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.555585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.555697 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.555752 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.556011 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.556089 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:57:55.556329 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.546791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.195:40387)
0810 14:57:55.546952 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.546993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.547475 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.547765 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.547771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.548147 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.548454 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.548463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.549418 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.549419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.550711 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.550714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.551184 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.551186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.551337 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.551833 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.551845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.553527 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.555039 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.555046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.555048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.555250 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.555426 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.555427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.555427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.555529 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.555750 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.555756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.555886 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.555979 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.556126 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260810 14:57:55.556779 32030 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34845
I20260810 14:57:55.556862 32030 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:55.557044 32030 heartbeater.cc:511] Master 127.28.193.254:34845 requested a full tablet report, sending...
I20260810 14:57:55.557428 31278 ts_manager.cc:194] Registered new tserver with Master: 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075)
I20260810 14:57:55.557865 31278 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:40387
I20260810 14:57:55.562053 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31816
I20260810 14:57:55.562144 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:57:55.562323 29447 external_mini_cluster.cc:1428] Setting key 8a168c896259520a2b03f7506b602acc
I20260810 14:57:55.562971 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.557787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:59221 (local address 127.28.193.254:34845)
0810 14:57:55.557825 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.557827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.557905 (+ 78us) server_negotiation.cc:407] Connection header received
0810 14:57:55.557973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.557974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.557984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.558030 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.558349 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.558935 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.559589 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.559775 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.559836 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.560668 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.560669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.560669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.560675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.561789 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.562165 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.562166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.562166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.562229 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.562446 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.562448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.562448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.562590 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.562663 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.562826 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.562907 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 14:57:55.562992 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.557732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.28.193.195:59221)
0810 14:57:55.557844 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.557853 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.557906 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.558035 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.558036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.558227 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:55.558279 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.558280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.559013 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.559014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.559504 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.559505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.559839 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.559840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.559913 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.560292 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:55.560296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:55.560591 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:55.561867 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.561869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.561869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.562078 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.562290 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:55.562291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:55.562291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:55.562358 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:55.562664 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:55.562665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:55.562715 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.562797 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.562898 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260810 14:57:55.563336 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:57:55 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:55.585407 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.579429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48878 (local address 127.28.193.254:34845)
0810 14:57:55.579493 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.579495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.579505 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:55.579530 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.579531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.579540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.579586 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:55.580071 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.580551 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.581210 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.581400 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.581453 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.583480 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:55.583482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:55.583482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:55.583488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:55.584635 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.584902 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.584903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.584903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.584946 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:55.585080 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:55.585080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:55.585080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:55.585163 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:55.585227 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.585294 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.585347 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:57:55.586946 31278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:57:55.588483 31278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:55.594542 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45875 (local address 127.0.0.1:39014)
0810 14:57:55.590686 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.590725 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.590789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.591113 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.591116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.591118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:55.591183 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.591185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592176 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.592177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.593272 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.593275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594240 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.594331 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594345 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.594411 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.594464 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260810 14:57:55.594585 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.28.193.193:45875)
0810 14:57:55.590908 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.590913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.590925 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:57:55.590971 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.590973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.590991 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.591278 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:55.591345 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592087 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.593381 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594153 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594227 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594285 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.594447 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.594488 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260810 14:57:55.594810 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36075 (local address 127.0.0.1:38096)
0810 14:57:55.591118 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.591128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.591186 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.591437 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.591438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.591439 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:55.591470 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.591471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592646 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.592647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.593527 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.593529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594475 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.594532 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594537 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.594580 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.594734 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":54,"thread_start_us":223,"threads_started":1}
I20260810 14:57:55.594980 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.28.193.194:42069)
0810 14:57:55.590981 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.590986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.591038 (+ 52us) server_negotiation.cc:407] Connection header received
0810 14:57:55.591135 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.591138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.591155 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.591228 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:55.591427 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592538 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.593832 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594505 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594566 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594612 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.594849 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.594891 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260810 14:57:55.595078 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.28.193.195:36075)
0810 14:57:55.591077 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.591081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.591187 (+ 106us) server_negotiation.cc:407] Connection header received
0810 14:57:55.591254 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.591256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.591272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.591336 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:55.591544 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592566 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.593609 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594319 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594388 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594437 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.594952 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.594993 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260810 14:57:55.595108 32059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.590570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42069 (local address 127.0.0.1:45932)
0810 14:57:55.590973 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.590984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.591050 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.591301 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.591302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.591304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:55.591353 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.591355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.592632 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.592633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.593706 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.593709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.594644 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.594645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.594705 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.594708 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.594749 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.594778 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":25,"thread_start_us":72,"threads_started":1}
I20260810 14:57:55.595286 31490 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6750aef643b0a625ca64dba49089 (DEFAULT_TABLE table=test-table [id=8bcc2d250edd41198a1b676218748579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.595712 31490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6750aef643b0a625ca64dba49089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.595892 31906 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6750aef643b0a625ca64dba49089 (DEFAULT_TABLE table=test-table [id=8bcc2d250edd41198a1b676218748579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.596213 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6750aef643b0a625ca64dba49089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.599522 32063 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Bootstrap starting.
I20260810 14:57:55.599813 32064 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Bootstrap starting.
I20260810 14:57:55.600046 31697 tablet_service.cc:1467] Processing CreateTablet for tablet d24c6750aef643b0a625ca64dba49089 (DEFAULT_TABLE table=test-table [id=8bcc2d250edd41198a1b676218748579]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.600405 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24c6750aef643b0a625ca64dba49089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.600445 32063 tablet_bootstrap.cc:654] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.600478 32064 tablet_bootstrap.cc:654] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.600723 32063 log.cc:824] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:55.600818 32064 log.cc:824] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:55.601562 32064 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: No bootstrap required, opened a new log
I20260810 14:57:55.601640 32064 ts_tablet_manager.cc:1397] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:55.603672 32064 raft_consensus.cc:348] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.603864 32064 raft_consensus.cc:374] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.603919 32064 raft_consensus.cc:729] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f58309c02741349922934827bea433, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.604079 32070 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Bootstrap starting.
I20260810 14:57:55.604072 32064 consensus_queue.cc:260] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.604357 32064 ts_tablet_manager.cc:1428] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 14:57:55.604396 31584 heartbeater.cc:503] Master 127.28.193.254:34845 was elected leader, sending a full tablet report...
I20260810 14:57:55.604763 32070 tablet_bootstrap.cc:654] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.604950 32063 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: No bootstrap required, opened a new log
I20260810 14:57:55.605019 32063 ts_tablet_manager.cc:1397] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 14:57:55.605116 32070 log.cc:824] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:55.605970 32070 tablet_bootstrap.cc:492] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: No bootstrap required, opened a new log
I20260810 14:57:55.606036 32070 ts_tablet_manager.cc:1397] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:55.606940 32063 raft_consensus.cc:348] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.607095 32063 raft_consensus.cc:374] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.607124 32063 raft_consensus.cc:729] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257f8081402546d5a98c3fa89f6b2cca, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.607226 32063 consensus_queue.cc:260] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.607471 32063 ts_tablet_manager.cc:1428] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 14:57:55.607494 32030 heartbeater.cc:503] Master 127.28.193.254:34845 was elected leader, sending a full tablet report...
I20260810 14:57:55.607839 32070 raft_consensus.cc:348] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.607995 32070 raft_consensus.cc:374] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.608039 32070 raft_consensus.cc:729] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7f01d6db724975adf662b0f713eb2a, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.608142 32070 consensus_queue.cc:260] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.608383 32070 ts_tablet_manager.cc:1428] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 14:57:55.608426 31801 heartbeater.cc:503] Master 127.28.193.254:34845 was elected leader, sending a full tablet report...
W20260810 14:57:55.646988 31804 tablet.cc:2369] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:57:55.745602 31585 tablet.cc:2369] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:57:55.804199 32031 tablet.cc:2369] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:57:55.911983 32073 raft_consensus.cc:482] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:55.912124 32073 raft_consensus.cc:504] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.912456 32073 leader_election.cc:288] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075), aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069)
I20260810 14:57:55.915772 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.912837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:50307 (local address 127.28.193.195:36075)
0810 14:57:55.912901 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.912904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.912917 (+ 13us) server_negotiation.cc:407] Connection header received
0810 14:57:55.913001 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.913002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.913013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.913070 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:55.913354 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.914081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915049 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.915432 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915505 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.915515 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.915655 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.915696 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:57:55.915781 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.912676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36075 (local address 127.28.193.193:50307)
0810 14:57:55.912845 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.912864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.912933 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.913101 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.913102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.913105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:55.913228 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.913231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.914160 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.914161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.914950 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.914953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.915512 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.915584 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.915588 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.915640 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.915684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260810 14:57:55.916085 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.912819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:51525 (local address 127.28.193.194:42069)
0810 14:57:55.912868 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.912870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.912882 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:57:55.913164 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.913166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.913177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.913237 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:55.914107 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.914574 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915405 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.915757 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915816 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.915823 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.915983 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.916021 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:57:55.916090 31943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6750aef643b0a625ca64dba49089" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257f8081402546d5a98c3fa89f6b2cca" is_pre_election: true
I20260810 14:57:55.916262 31943 raft_consensus.cc:2454] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 0.
I20260810 14:57:55.916291 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.912666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42069 (local address 127.28.193.193:51525)
0810 14:57:55.912831 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:55.912843 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:55.913042 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:55.913276 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:55.913277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:55.913280 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:55.913388 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.913391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.914655 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.914656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.915324 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:55.915326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.915825 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.915826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:55.915892 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.915896 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:57:55.916175 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 14:57:55.916219 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260810 14:57:55.916545 31411 leader_election.cc:302] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257f8081402546d5a98c3fa89f6b2cca, 37f58309c02741349922934827bea433; no voters:
I20260810 14:57:55.916594 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6750aef643b0a625ca64dba49089" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a" is_pre_election: true
I20260810 14:57:55.916692 32073 raft_consensus.cc:2781] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:55.916772 32073 raft_consensus.cc:482] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:55.916759 31721 raft_consensus.cc:2454] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 0.
I20260810 14:57:55.916807 32073 raft_consensus.cc:3037] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.917557 32073 raft_consensus.cc:504] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.917685 32073 leader_election.cc:288] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 election: Requested vote from peers 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075), aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069)
I20260810 14:57:55.917853 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6750aef643b0a625ca64dba49089" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a"
I20260810 14:57:55.917860 31943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24c6750aef643b0a625ca64dba49089" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257f8081402546d5a98c3fa89f6b2cca"
I20260810 14:57:55.917939 31721 raft_consensus.cc:3037] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.917939 31943 raft_consensus.cc:3037] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.918883 31721 raft_consensus.cc:2454] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 1.
I20260810 14:57:55.918883 31943 raft_consensus.cc:2454] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 1.
I20260810 14:57:55.919044 31415 leader_election.cc:302] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37f58309c02741349922934827bea433, aa7f01d6db724975adf662b0f713eb2a; no voters:
I20260810 14:57:55.919170 32073 raft_consensus.cc:2781] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:55.919313 32073 raft_consensus.cc:686] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 1 LEADER]: Becoming Leader. State: Replica: 37f58309c02741349922934827bea433, State: Running, Role: LEADER
I20260810 14:57:55.919438 32073 consensus_queue.cc:237] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } }
I20260810 14:57:55.920250 31278 catalog_manager.cc:5495] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37f58309c02741349922934827bea433 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "37f58309c02741349922934827bea433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } health_report { overall_health: UNKNOWN } } }
I20260810 14:57:55.946348 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:55.943992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.28.193.193:45875)
0810 14:57:55.944059 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:55.944062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:55.944072 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:55.944114 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:55.944115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:55.944137 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:55.944193 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:55.944334 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.944808 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.945582 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:55.945792 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:55.945848 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:55.946040 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:55.946135 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:55.946204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:55.946250 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 14:57:55.946273 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:57:55.948129 31721 raft_consensus.cc:1261] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Refusing update from remote peer 37f58309c02741349922934827bea433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:55.948129 31943 raft_consensus.cc:1261] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Refusing update from remote peer 37f58309c02741349922934827bea433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:55.948480 32083 consensus_queue.cc:1035] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:55.948618 32073 consensus_queue.cc:1035] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:55.951174 32091 mvcc.cc:205] Tried to move back new op lower bound from 7316987395880775680 to 7316987395766591488. Current Snapshot: MvccSnapshot[applied={T|T < 7316987395880775680}]
I20260810 14:57:55.955108 31272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:55.955204 31272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:55.956004 31272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Sending DeleteTablet for 3 replicas of tablet d24c6750aef643b0a625ca64dba49089
I20260810 14:57:55.956442 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6750aef643b0a625ca64dba49089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:38096
I20260810 14:57:55.956682 32106 tablet_replica.cc:330] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: stopping tablet replica
I20260810 14:57:55.956629 31272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:57:55.956687 31490 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6750aef643b0a625ca64dba49089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:39014
W20260810 14:57:55.956768 31272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:55.956771 32106 raft_consensus.cc:2229] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:57:55.956871 32106 raft_consensus.cc:2258] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:55.956923 31697 tablet_service.cc:1514] Processing DeleteTablet for tablet d24c6750aef643b0a625ca64dba49089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:55 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:45932
I20260810 14:57:55.957100 32108 tablet_replica.cc:330] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: stopping tablet replica
I20260810 14:57:55.957231 32108 raft_consensus.cc:2229] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:57:55.957327 32108 raft_consensus.cc:2258] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:55.957619 32107 tablet_replica.cc:330] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: stopping tablet replica
I20260810 14:57:55.957796 32107 raft_consensus.cc:2229] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:57:55.957805 32108 ts_tablet_manager.cc:1905] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:55.957989 32107 raft_consensus.cc:2258] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:55.958429 32107 ts_tablet_manager.cc:1905] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:55.958534 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c54aa967943d29118a34cb48723a3 (DEFAULT_TABLE table=test-table [id=1fb92e539edc4352b48f00c0d66adb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.958655 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c54aa967943d29118a34cb48723a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.959563 32108 ts_tablet_manager.cc:1918] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:57:55.959635 32108 log.cc:1196] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/d24c6750aef643b0a625ca64dba49089
I20260810 14:57:55.959903 32108 ts_tablet_manager.cc:1939] T d24c6750aef643b0a625ca64dba49089 P aa7f01d6db724975adf662b0f713eb2a: Deleting consensus metadata
I20260810 14:57:55.960240 31906 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c54aa967943d29118a34cb48723a3 (DEFAULT_TABLE table=test-table [id=1fb92e539edc4352b48f00c0d66adb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.960361 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c54aa967943d29118a34cb48723a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.960704 32107 ts_tablet_manager.cc:1918] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:57:55.960788 32107 log.cc:1196] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/d24c6750aef643b0a625ca64dba49089
I20260810 14:57:55.961077 32107 ts_tablet_manager.cc:1939] T d24c6750aef643b0a625ca64dba49089 P 37f58309c02741349922934827bea433: Deleting consensus metadata
I20260810 14:57:55.961459 32063 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Bootstrap starting.
I20260810 14:57:55.961474 32070 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Bootstrap starting.
I20260810 14:57:55.961875 32063 tablet_bootstrap.cc:654] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.961875 32070 tablet_bootstrap.cc:654] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.962283 31258 catalog_manager.cc:4841] TS aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069): tablet d24c6750aef643b0a625ca64dba49089 (table test-table [id=8bcc2d250edd41198a1b676218748579]) successfully deleted
I20260810 14:57:55.962558 31490 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1c54aa967943d29118a34cb48723a3 (DEFAULT_TABLE table=test-table [id=1fb92e539edc4352b48f00c0d66adb46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:55.962658 31490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d1c54aa967943d29118a34cb48723a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:55.962690 32063 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: No bootstrap required, opened a new log
I20260810 14:57:55.962751 32063 ts_tablet_manager.cc:1397] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:55.962920 32063 raft_consensus.cc:348] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.963023 32063 raft_consensus.cc:374] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.963047 32063 raft_consensus.cc:729] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257f8081402546d5a98c3fa89f6b2cca, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.963104 32063 consensus_queue.cc:260] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.963207 32063 ts_tablet_manager.cc:1428] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 14:57:55.963701 32064 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Bootstrap starting.
I20260810 14:57:55.963861 31257 catalog_manager.cc:4841] TS 37f58309c02741349922934827bea433 (127.28.193.193:45875): tablet d24c6750aef643b0a625ca64dba49089 (table test-table [id=8bcc2d250edd41198a1b676218748579]) successfully deleted
I20260810 14:57:55.964113 32064 tablet_bootstrap.cc:654] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:55.964229 32070 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: No bootstrap required, opened a new log
I20260810 14:57:55.964268 32070 ts_tablet_manager.cc:1397] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 14:57:55.964622 32070 raft_consensus.cc:348] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.964704 32070 raft_consensus.cc:374] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.964723 32070 raft_consensus.cc:729] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7f01d6db724975adf662b0f713eb2a, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.964777 32064 tablet_bootstrap.cc:492] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: No bootstrap required, opened a new log
I20260810 14:57:55.964766 32070 consensus_queue.cc:260] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.964813 32064 ts_tablet_manager.cc:1397] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:55.964864 32070 ts_tablet_manager.cc:1428] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:57:55.964946 32064 raft_consensus.cc:348] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.965014 32064 raft_consensus.cc:374] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:55.965031 32064 raft_consensus.cc:729] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f58309c02741349922934827bea433, State: Initialized, Role: FOLLOWER
I20260810 14:57:55.965070 32064 consensus_queue.cc:260] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.965147 32064 ts_tablet_manager.cc:1428] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:55.965287 32106 ts_tablet_manager.cc:1905] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:55.966727 32106 ts_tablet_manager.cc:1918] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:57:55.966799 32106 log.cc:1196] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/d24c6750aef643b0a625ca64dba49089
I20260810 14:57:55.967047 32106 ts_tablet_manager.cc:1939] T d24c6750aef643b0a625ca64dba49089 P 257f8081402546d5a98c3fa89f6b2cca: Deleting consensus metadata
I20260810 14:57:55.967427 31255 catalog_manager.cc:4841] TS 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075): tablet d24c6750aef643b0a625ca64dba49089 (table test-table [id=8bcc2d250edd41198a1b676218748579]) successfully deleted
I20260810 14:57:55.991374 32083 raft_consensus.cc:482] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:55.991504 32083 raft_consensus.cc:504] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.991695 32083 leader_election.cc:288] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069), 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075)
I20260810 14:57:55.991879 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c54aa967943d29118a34cb48723a3" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a" is_pre_election: true
I20260810 14:57:55.991899 31943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c54aa967943d29118a34cb48723a3" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257f8081402546d5a98c3fa89f6b2cca" is_pre_election: true
I20260810 14:57:55.991984 31721 raft_consensus.cc:2454] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 0.
I20260810 14:57:55.991995 31943 raft_consensus.cc:2454] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 0.
I20260810 14:57:55.992132 31415 leader_election.cc:302] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37f58309c02741349922934827bea433, aa7f01d6db724975adf662b0f713eb2a; no voters:
I20260810 14:57:55.992234 32083 raft_consensus.cc:2781] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:55.992267 32083 raft_consensus.cc:482] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:55.992285 32083 raft_consensus.cc:3037] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.992973 32083 raft_consensus.cc:504] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.993080 32083 leader_election.cc:288] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 election: Requested vote from peers aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069), 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075)
I20260810 14:57:55.993268 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c54aa967943d29118a34cb48723a3" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a"
I20260810 14:57:55.993261 31943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1c54aa967943d29118a34cb48723a3" candidate_uuid: "37f58309c02741349922934827bea433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "257f8081402546d5a98c3fa89f6b2cca"
I20260810 14:57:55.993335 31943 raft_consensus.cc:3037] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.993335 31721 raft_consensus.cc:3037] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:55.994089 31943 raft_consensus.cc:2454] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 1.
I20260810 14:57:55.994123 31721 raft_consensus.cc:2454] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f58309c02741349922934827bea433 in term 1.
I20260810 14:57:55.994254 31411 leader_election.cc:302] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257f8081402546d5a98c3fa89f6b2cca, 37f58309c02741349922934827bea433; no voters:
I20260810 14:57:55.994390 32083 raft_consensus.cc:2781] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:55.994434 32083 raft_consensus.cc:686] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 1 LEADER]: Becoming Leader. State: Replica: 37f58309c02741349922934827bea433, State: Running, Role: LEADER
I20260810 14:57:55.994486 32083 consensus_queue.cc:237] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:55.995064 31272 catalog_manager.cc:5495] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37f58309c02741349922934827bea433 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "37f58309c02741349922934827bea433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:57:56 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:56.041177 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.030595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.28.193.254:34845)
0810 14:57:56.030661 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.030663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.030785 (+ 122us) server_negotiation.cc:407] Connection header received
0810 14:57:56.030862 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.030863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.030872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.030922 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:56.031514 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.031995 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.033007 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.033192 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.033251 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.038738 (+ 5487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:56.038739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:56.038739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:56.038745 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:56.040061 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.040499 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.040499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.040500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.040543 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.040734 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.040734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.040735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.040825 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:56.040867 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.041045 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.041110 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:57:56.041299 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.030524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34845 (local address 127.0.0.1:48884)
0810 14:57:56.030705 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.030718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.030800 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.030948 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.030949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.031371 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:56.031435 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.031439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.032077 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.032078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.032911 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.032915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.033271 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.033272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.033334 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.036968 (+ 3634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:56.036974 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:56.038650 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:56.040149 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:56.040158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:56.040161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:56.040427 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:56.040602 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:56.040602 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:56.040603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:56.040677 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:56.040880 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:56.040887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:56.040950 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.041029 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.041187 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":39,"thread_start_us":95,"threads_started":1}
I20260810 14:57:56.043658 31272 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:48884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:57:56.043879 31272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:56.045285 31906 tablet_service.cc:1467] Processing CreateTablet for tablet a55fd1162c2749c29460ebde02b3f004 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d400cfa0c7432f97bc791bf7b4b1c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:56.045295 31697 tablet_service.cc:1467] Processing CreateTablet for tablet a55fd1162c2749c29460ebde02b3f004 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d400cfa0c7432f97bc791bf7b4b1c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:56.045327 31490 tablet_service.cc:1467] Processing CreateTablet for tablet a55fd1162c2749c29460ebde02b3f004 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0d400cfa0c7432f97bc791bf7b4b1c1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:56.045444 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55fd1162c2749c29460ebde02b3f004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:56.045449 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55fd1162c2749c29460ebde02b3f004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:56.045454 31490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55fd1162c2749c29460ebde02b3f004. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:56.046988 32064 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: Bootstrap starting.
I20260810 14:57:56.047314 32064 tablet_bootstrap.cc:654] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:56.047925 32064 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: No bootstrap required, opened a new log
I20260810 14:57:56.047959 32064 ts_tablet_manager.cc:1397] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:56.048084 32064 raft_consensus.cc:348] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.048136 32064 raft_consensus.cc:374] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:56.048148 32064 raft_consensus.cc:729] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f58309c02741349922934827bea433, State: Initialized, Role: FOLLOWER
I20260810 14:57:56.048179 32064 consensus_queue.cc:260] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.048234 32064 ts_tablet_manager.cc:1428] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:56.048307 32083 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.048343 32083 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.049145 32070 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: Bootstrap starting.
I20260810 14:57:56.049489 32063 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: Bootstrap starting.
I20260810 14:57:56.049577 32070 tablet_bootstrap.cc:654] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:56.049991 32063 tablet_bootstrap.cc:654] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:56.050209 32070 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: No bootstrap required, opened a new log
I20260810 14:57:56.050256 32070 ts_tablet_manager.cc:1397] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:56.050402 32070 raft_consensus.cc:348] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.050483 32070 raft_consensus.cc:374] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:56.050532 32070 raft_consensus.cc:729] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7f01d6db724975adf662b0f713eb2a, State: Initialized, Role: FOLLOWER
I20260810 14:57:56.050573 32070 consensus_queue.cc:260] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.050664 32070 ts_tablet_manager.cc:1428] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:56.050688 32063 tablet_bootstrap.cc:492] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: No bootstrap required, opened a new log
I20260810 14:57:56.050733 32063 ts_tablet_manager.cc:1397] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:56.050714 32076 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.050782 32076 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.050879 32063 raft_consensus.cc:348] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.050973 32063 raft_consensus.cc:374] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:56.051000 32063 raft_consensus.cc:729] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257f8081402546d5a98c3fa89f6b2cca, State: Initialized, Role: FOLLOWER
I20260810 14:57:56.051048 32063 consensus_queue.cc:260] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.051141 32063 ts_tablet_manager.cc:1428] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:56.051395 32075 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.051482 32075 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.216905 32075 raft_consensus.cc:482] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:56.217022 32075 raft_consensus.cc:504] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.217383 32075 leader_election.cc:288] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f58309c02741349922934827bea433 (127.28.193.193:45875), aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069)
I20260810 14:57:56.221192 32061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.217748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:46135 (local address 127.28.193.194:42069)
0810 14:57:56.217789 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.217791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.217801 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:56.217862 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.217864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.217873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.217936 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:56.218097 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.218848 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220284 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.220838 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220912 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.220922 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.221074 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.221113 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:57:56.221192 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.217658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44833 (local address 127.28.193.193:45875)
0810 14:57:56.217731 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.217734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.217797 (+ 63us) server_negotiation.cc:407] Connection header received
0810 14:57:56.217865 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.217866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.217881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.217961 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:56.218098 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.218850 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220285 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.220842 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220913 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.220930 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.221074 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.221112 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:57:56.221231 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.217574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45875 (local address 127.28.193.195:44833)
0810 14:57:56.217726 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.217738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.217801 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.217966 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.217967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.217968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:56.218030 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.218032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.218925 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.218926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.220195 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.220198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.220917 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.221004 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.221009 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.221064 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.221118 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":62,"thread_start_us":73,"threads_started":1}
I20260810 14:57:56.221240 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.217582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42069 (local address 127.28.193.195:46135)
0810 14:57:56.217730 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.217738 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.217801 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.217950 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.217951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.217953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:56.218034 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.218036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.218930 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.218931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.220195 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.220198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.220920 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.220920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.221008 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.221010 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.221064 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.221118 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":47,"thread_start_us":68,"threads_started":1}
I20260810 14:57:56.221678 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55fd1162c2749c29460ebde02b3f004" candidate_uuid: "257f8081402546d5a98c3fa89f6b2cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a" is_pre_election: true
I20260810 14:57:56.221678 31514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55fd1162c2749c29460ebde02b3f004" candidate_uuid: "257f8081402546d5a98c3fa89f6b2cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f58309c02741349922934827bea433" is_pre_election: true
I20260810 14:57:56.221797 31721 raft_consensus.cc:2454] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 257f8081402546d5a98c3fa89f6b2cca in term 0.
I20260810 14:57:56.221817 31514 raft_consensus.cc:2454] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 257f8081402546d5a98c3fa89f6b2cca in term 0.
I20260810 14:57:56.221975 31841 leader_election.cc:302] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257f8081402546d5a98c3fa89f6b2cca, aa7f01d6db724975adf662b0f713eb2a; no voters:
I20260810 14:57:56.222114 32075 raft_consensus.cc:2781] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:56.222152 32075 raft_consensus.cc:482] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:56.222172 32075 raft_consensus.cc:3037] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:56.222880 32075 raft_consensus.cc:504] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.223014 32075 leader_election.cc:288] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [CANDIDATE]: Term 1 election: Requested vote from peers 37f58309c02741349922934827bea433 (127.28.193.193:45875), aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069)
I20260810 14:57:56.223162 31514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55fd1162c2749c29460ebde02b3f004" candidate_uuid: "257f8081402546d5a98c3fa89f6b2cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f58309c02741349922934827bea433"
I20260810 14:57:56.223151 31721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55fd1162c2749c29460ebde02b3f004" candidate_uuid: "257f8081402546d5a98c3fa89f6b2cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7f01d6db724975adf662b0f713eb2a"
I20260810 14:57:56.223233 31514 raft_consensus.cc:3037] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:56.223237 31721 raft_consensus.cc:3037] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:56.223812 31721 raft_consensus.cc:2454] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 257f8081402546d5a98c3fa89f6b2cca in term 1.
I20260810 14:57:56.223894 31514 raft_consensus.cc:2454] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 257f8081402546d5a98c3fa89f6b2cca in term 1.
I20260810 14:57:56.224001 31841 leader_election.cc:302] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 257f8081402546d5a98c3fa89f6b2cca, aa7f01d6db724975adf662b0f713eb2a; no voters:
I20260810 14:57:56.224097 32075 raft_consensus.cc:2781] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:56.224231 32075 raft_consensus.cc:686] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [term 1 LEADER]: Becoming Leader. State: Replica: 257f8081402546d5a98c3fa89f6b2cca, State: Running, Role: LEADER
I20260810 14:57:56.224305 32075 consensus_queue.cc:237] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } }
I20260810 14:57:56.224607 32137 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: TxnStatusTablet state changed. Reason: New leader 257f8081402546d5a98c3fa89f6b2cca. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.224709 32137 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:56.225023 31272 catalog_manager.cc:5495] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca reported cstate change: term changed from 0 to 1, leader changed from <none> to 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195). New cstate: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } health_report { overall_health: HEALTHY } } }
I20260810 14:57:56.225323 32142 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:57:56.225557 32142 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:57:56.233541 32121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.230716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36075 (local address 127.0.0.1:38098)
0810 14:57:56.230785 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.230798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.230846 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.231013 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.231014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.231014 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:56.231059 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.231062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.231940 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.231941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.232538 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.232540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.232993 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.232994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.233051 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.233070 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.233372 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.233385 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.233435 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.233483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 14:57:56.233553 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.230814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.28.193.195:36075)
0810 14:57:56.230867 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.230869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.230879 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:56.230912 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.230913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.230931 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.231013 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:56.231110 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.231833 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.232620 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.232909 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.232982 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.233160 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.233281 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.233358 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.233455 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.233477 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:57:56.234864 31721 raft_consensus.cc:1261] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Refusing update from remote peer 257f8081402546d5a98c3fa89f6b2cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:56.234948 31514 raft_consensus.cc:1261] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Refusing update from remote peer 257f8081402546d5a98c3fa89f6b2cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:56.235159 32075 consensus_queue.cc:1035] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:56.235294 32137 consensus_queue.cc:1035] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:56.235924 32083 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: TxnStatusTablet state changed. Reason: New leader 257f8081402546d5a98c3fa89f6b2cca. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.235953 32076 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: TxnStatusTablet state changed. Reason: New leader 257f8081402546d5a98c3fa89f6b2cca. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.236016 32083 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.236024 32076 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.237012 32075 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.237130 32075 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:56.237138 32088 mvcc.cc:205] Tried to move back new op lower bound from 7316987397055934464 to 7316987397015404544. Current Snapshot: MvccSnapshot[applied={T|T < 7316987397055934464}]
I20260810 14:57:56.237197 32092 mvcc.cc:205] Tried to move back new op lower bound from 7316987397055934464 to 7316987397015404544. Current Snapshot: MvccSnapshot[applied={T|T < 7316987397055934464}]
I20260810 14:57:56.237228 32083 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.237299 32083 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 37f58309c02741349922934827bea433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.238054 32076 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.238143 32076 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P aa7f01d6db724975adf662b0f713eb2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:56.237017 32137 tablet_replica.cc:439] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "257f8081402546d5a98c3fa89f6b2cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f58309c02741349922934827bea433" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45875 } } peers { permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 } } peers { permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 } } }
I20260810 14:57:56.238713 32137 tablet_replica.cc:442] T a55fd1162c2749c29460ebde02b3f004 P 257f8081402546d5a98c3fa89f6b2cca: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:56.243304 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.241123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:42911 (local address 127.28.193.195:36075)
0810 14:57:56.241163 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.241164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.241282 (+ 118us) server_negotiation.cc:407] Connection header received
0810 14:57:56.241344 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.241345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.241355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.241395 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:56.241490 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.241985 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.242705 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.242883 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.242936 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.243096 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.243124 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.243164 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.243213 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.243232 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 14:57:56.243327 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.241072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36075 (local address 127.28.193.193:42911)
0810 14:57:56.241220 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.241231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.241288 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.241396 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.241396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.241398 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:56.241446 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.241448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.242059 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.242060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.242636 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.242638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.242952 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.242956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.243016 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.243029 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.243170 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.243172 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.243212 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.243258 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260810 14:57:56.248088 31943 raft_consensus.cc:1261] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Refusing update from remote peer 37f58309c02741349922934827bea433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:56.248168 31719 raft_consensus.cc:1261] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Refusing update from remote peer 37f58309c02741349922934827bea433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:56.248282 32073 consensus_queue.cc:1035] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "257f8081402546d5a98c3fa89f6b2cca" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:56.248378 32083 consensus_queue.cc:1035] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7f01d6db724975adf662b0f713eb2a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:56.258262 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.255132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:58297 (local address 127.28.193.193:45875)
0810 14:57:56.255171 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.255173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.255183 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:56.255257 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.255258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.255272 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.255336 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:56.255432 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.256200 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.257212 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.257527 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.257590 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.257982 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.258020 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.258047 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.258162 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.258192 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:57:56.258297 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.254898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45875 (local address 127.28.193.195:58297)
0810 14:57:56.255106 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:56.255118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:56.255205 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:56.255338 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:56.255339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:56.255340 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:56.255395 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.255396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.256289 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.256291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.257129 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:56.257152 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.257642 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.257673 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:56.257837 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.257856 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:56.258085 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:56.258087 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:56.258146 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 14:57:56.258190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":126,"threads_started":1}
I20260810 14:57:56.264830 31272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:56.264928 31272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:56.265486 31272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38bd5582b9bc4392bdf98b28650cab57: Sending DeleteTablet for 3 replicas of tablet 3d1c54aa967943d29118a34cb48723a3
I20260810 14:57:56.265750 31697 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1c54aa967943d29118a34cb48723a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:56 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:45932
I20260810 14:57:56.265774 31490 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1c54aa967943d29118a34cb48723a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:56 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:39014
I20260810 14:57:56.265811 31906 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d1c54aa967943d29118a34cb48723a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:56 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:38096
I20260810 14:57:56.265848 32108 tablet_replica.cc:330] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: stopping tablet replica
I20260810 14:57:56.265869 32107 tablet_replica.cc:330] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: stopping tablet replica
I20260810 14:57:56.265882 32106 tablet_replica.cc:330] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: stopping tablet replica
I20260810 14:57:56.265892 32108 raft_consensus.cc:2229] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:57:56.265914 32107 raft_consensus.cc:2229] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:57:56.265920 32106 raft_consensus.cc:2229] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:57:56.265925 32108 raft_consensus.cc:2258] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:56.265951 32106 raft_consensus.cc:2258] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:56.265962 32107 raft_consensus.cc:2258] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:56.266330 32106 ts_tablet_manager.cc:1905] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:56.266373 32108 ts_tablet_manager.cc:1905] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:56.266392 32107 ts_tablet_manager.cc:1905] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:56.267941 32107 ts_tablet_manager.cc:1918] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:56.267968 32106 ts_tablet_manager.cc:1918] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:56.268003 32107 log.cc:1196] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/3d1c54aa967943d29118a34cb48723a3
I20260810 14:57:56.267997 32108 ts_tablet_manager.cc:1918] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:56.268021 32106 log.cc:1196] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/3d1c54aa967943d29118a34cb48723a3
I20260810 14:57:56.268040 32108 log.cc:1196] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/3d1c54aa967943d29118a34cb48723a3
I20260810 14:57:56.268219 32107 ts_tablet_manager.cc:1939] T 3d1c54aa967943d29118a34cb48723a3 P 37f58309c02741349922934827bea433: Deleting consensus metadata
I20260810 14:57:56.268235 32106 ts_tablet_manager.cc:1939] T 3d1c54aa967943d29118a34cb48723a3 P 257f8081402546d5a98c3fa89f6b2cca: Deleting consensus metadata
I20260810 14:57:56.268235 32108 ts_tablet_manager.cc:1939] T 3d1c54aa967943d29118a34cb48723a3 P aa7f01d6db724975adf662b0f713eb2a: Deleting consensus metadata
I20260810 14:57:56.268532 31257 catalog_manager.cc:4841] TS 37f58309c02741349922934827bea433 (127.28.193.193:45875): tablet 3d1c54aa967943d29118a34cb48723a3 (table test-table [id=1fb92e539edc4352b48f00c0d66adb46]) successfully deleted
I20260810 14:57:56.268595 31258 catalog_manager.cc:4841] TS aa7f01d6db724975adf662b0f713eb2a (127.28.193.194:42069): tablet 3d1c54aa967943d29118a34cb48723a3 (table test-table [id=1fb92e539edc4352b48f00c0d66adb46]) successfully deleted
I20260810 14:57:56.268595 31255 catalog_manager.cc:4841] TS 257f8081402546d5a98c3fa89f6b2cca (127.28.193.195:36075): tablet 3d1c54aa967943d29118a34cb48723a3 (table test-table [id=1fb92e539edc4352b48f00c0d66adb46]) successfully deleted
Aug 10 14:57:56 dist-test-slave-cvwc krb5kdc[31210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:57:56.278827 32060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.270009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.28.193.193:45875)
0810 14:57:56.270060 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.270062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.270152 (+ 90us) server_negotiation.cc:407] Connection header received
0810 14:57:56.270191 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.270192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.270204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.270268 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:56.270814 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.271558 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.272334 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.272647 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.272715 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.275899 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:56.275912 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:56.275912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:56.275920 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:56.277594 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.278016 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.278019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.278019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.278078 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.278275 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.278275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.278275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.278440 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:56.278556 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.278641 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.278708 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 14:57:56.279131 31574 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39018
I20260810 14:57:56.285650 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:56.280109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.28.193.254:34845)
0810 14:57:56.280159 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:56.280161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:56.280190 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:57:56.280265 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:56.280266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:56.280276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:56.280331 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:56.280792 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.281270 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.281956 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:56.282156 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:56.282209 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:56.283281 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:56.283283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:56.283283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:56.283291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:56.284531 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.284850 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.284851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.284852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.284910 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:56.285178 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:56.285179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:56.285180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:56.285297 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:56.285384 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:56.285486 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 14:57:56.285562 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 14:57:56.285883 31272 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48894
I20260810 14:57:56.286471 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31387
I20260810 14:57:56.307842 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31597
I20260810 14:57:56.322656 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31816
I20260810 14:57:56.334493 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31226
2026-08-10T14:57:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2699 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:57:56 dist-test-slave-cvwc krb5kdc[32174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:57:56 dist-test-slave-cvwc krb5kdc[32174](info): set up 2 sockets
Aug 10 14:57:56 dist-test-slave-cvwc krb5kdc[32174](info): commencing operation
krb5kdc: starting...
W20260810 14:57:57.505137 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:57:57 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:57:57.621670 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35345
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:57.703861 32194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:57.704088 32194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:57.704118 32194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:57.705246 32194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:57:57.705286 32194 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:57:57.705303 32194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:57.705319 32194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:57.705334 32194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:57:57.705350 32194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:57:57.706815 32194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:35345
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:57.707064 32194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:57.707265 32194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:57:57.712054 32194 server_base.cc:1034] running on GCE node
W20260810 14:57:57.711984 32209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:57.711985 32203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:57.712054 32200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:57.712447 32194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:57.712728 32194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:57.713855 32194 hybrid_clock.cc:648] HybridClock initialized: now 1786373877713841 us; error 30 us; skew 500 ppm
Aug 10 14:57:57 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:57.716380 32194 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:57:57.716816 32194 webserver.cc:460] Webserver started at http://127.28.193.254:39613/ using document root <none> and password file <none>
I20260810 14:57:57.717013 32194 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:57.717056 32194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:57.717172 32194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:57.718227 32194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "c1d54faf77b24a36ad253f01cb537c8a"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "604e11867c536c0fa9f202001eb6be21"
server_key_iv: "0309866c156fed2598f4dc6dd8244784"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.718582 32194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "c1d54faf77b24a36ad253f01cb537c8a"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "604e11867c536c0fa9f202001eb6be21"
server_key_iv: "0309866c156fed2598f4dc6dd8244784"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.719825 32194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:57.720453 32219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:57.720597 32194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:57.720680 32194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "c1d54faf77b24a36ad253f01cb537c8a"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "604e11867c536c0fa9f202001eb6be21"
server_key_iv: "0309866c156fed2598f4dc6dd8244784"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.720739 32194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:57.749651 32194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:57.751854 32194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:57.751999 32194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:57.756286 32194 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:35345
I20260810 14:57:57.756367 32323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:35345 every 8 connection(s)
I20260810 14:57:57.756667 32194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:57:57.757192 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32194
I20260810 14:57:57.757275 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:57:57.757352 32324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:57.757452 29447 external_mini_cluster.cc:1428] Setting key 4a643bac5679462583d8282a349c940b
I20260810 14:57:57.760624 32324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Bootstrap starting.
I20260810 14:57:57.761292 32324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:57.761569 32324 log.cc:824] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:57.762256 32324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: No bootstrap required, opened a new log
Aug 10 14:57:57 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:57.764562 32324 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } }
I20260810 14:57:57.764706 32324 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:57.764741 32324 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d54faf77b24a36ad253f01cb537c8a, State: Initialized, Role: FOLLOWER
I20260810 14:57:57.764879 32324 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } }
I20260810 14:57:57.764981 32324 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:57:57.765013 32324 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:57:57.765062 32324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:57.765699 32324 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } }
I20260810 14:57:57.765831 32324 leader_election.cc:302] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1d54faf77b24a36ad253f01cb537c8a; no voters:
I20260810 14:57:57.765998 32324 leader_election.cc:288] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:57:57.766036 32330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:57.766181 32330 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [term 1 LEADER]: Becoming Leader. State: Replica: c1d54faf77b24a36ad253f01cb537c8a, State: Running, Role: LEADER
I20260810 14:57:57.766254 32324 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:57:57.766284 32330 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } }
I20260810 14:57:57.766615 32334 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1d54faf77b24a36ad253f01cb537c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } } }
I20260810 14:57:57.766613 32330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1d54faf77b24a36ad253f01cb537c8a. Latest consensus state: current_term: 1 leader_uuid: "c1d54faf77b24a36ad253f01cb537c8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1d54faf77b24a36ad253f01cb537c8a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35345 } } }
I20260810 14:57:57.766778 32334 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:57.766796 32330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a [sys.catalog]: This master's current role is: LEADER
I20260810 14:57:57.767119 32345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:57:57.767815 32345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:57:57.768911 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:57.757971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.28.193.254:35345)
0810 14:57:57.758161 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:57.758171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:57.758208 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:57:57.758869 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:57.758880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:57.758955 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:57.759129 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:57.759685 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.761043 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.761915 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.762320 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.762420 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:57.764855 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:57.764859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:57.764860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:57.764871 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:57.766685 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.767116 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.767119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.767120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.767242 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.767464 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.767465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.767466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.767782 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:57.767916 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:57.768586 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 14:57:57.768718 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260810 14:57:57.769680 32345 catalog_manager.cc:1347] Generated new cluster ID: c48f123b292d43d5b5040d940fa47347
I20260810 14:57:57.769754 32345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:57:57.775368 32345 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:57:57.776059 32345 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:57:57.781975 32345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Generated new TSK 0
I20260810 14:57:57.782143 32345 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:57:57.809832 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35345
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:57.892882 32361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:57.893105 32361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:57.893131 32361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:57.894361 32361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:57.894409 32361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:57.894429 32361 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:57:57.894440 32361 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:57.894456 32361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:57:57.895942 32361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:57.896185 32361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:57.896413 32361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:57:57.901466 32361 server_base.cc:1034] running on GCE node
W20260810 14:57:57.901397 32373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:57.901412 32377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:57.901397 32370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:57.901809 32361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:57.902070 32361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:57.903220 32361 hybrid_clock.cc:648] HybridClock initialized: now 1786373877903208 us; error 27 us; skew 500 ppm
Aug 10 14:57:57 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:57.905840 32361 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:57:57.906306 32361 webserver.cc:460] Webserver started at http://127.28.193.193:40915/ using document root <none> and password file <none>
I20260810 14:57:57.906551 32361 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:57.906599 32361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:57.906721 32361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:57.907769 32361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "7f59298cad414adb9877300d39aff455"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "03fe98c4db9be898ab7ac534daceb67e"
server_key_iv: "36f99bdb27f04ea497fd816ceaf2ef82"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.908114 32361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "7f59298cad414adb9877300d39aff455"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "03fe98c4db9be898ab7ac534daceb67e"
server_key_iv: "36f99bdb27f04ea497fd816ceaf2ef82"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.909386 32361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 14:57:57.910216 32388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:57.910409 32361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 14:57:57.910501 32361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "7f59298cad414adb9877300d39aff455"
format_stamp: "Formatted at 2026-08-10 14:57:57 on dist-test-slave-cvwc"
server_key: "03fe98c4db9be898ab7ac534daceb67e"
server_key_iv: "36f99bdb27f04ea497fd816ceaf2ef82"
server_key_version: "encryptionkey@0"
I20260810 14:57:57.910575 32361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:57.924717 32361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:57.926901 32361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:57.927033 32361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:57.927588 32361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:57.927635 32361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:57.927656 32361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:57.927668 32361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:57:57 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:57.934269 32361 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:41669
I20260810 14:57:57.934365 32578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:41669 every 8 connection(s)
I20260810 14:57:57.934690 32361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:57:57.937047 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:57.928076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54005 (local address 127.28.193.254:35345)
0810 14:57:57.928145 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:57.928148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:57.928350 (+ 202us) server_negotiation.cc:407] Connection header received
0810 14:57:57.928875 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:57.928876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:57.928891 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:57.928969 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:57.929848 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.930607 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.931646 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.931962 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.932039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:57.934804 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:57.934806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:57.934806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:57.934814 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:57.935997 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.936343 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.936344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.936344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.936392 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.936579 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.936579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.936580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.936676 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:57.936722 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:57.936920 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 14:57:57.936984 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:57:57.937227 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:57.928046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.193:54005)
0810 14:57:57.928213 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:57.928253 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:57.928754 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:57.929051 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:57.929057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:57.929410 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:57.929744 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:57.929750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.930707 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.930708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:57.931577 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:57.931579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.932056 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.932057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:57.932201 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:57.932798 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:57.932806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:57.934705 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:57.936068 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:57.936073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:57.936075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:57.936286 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:57.936445 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:57.936446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:57.936446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:57.936537 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:57.936717 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:57.936720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:57.936832 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:57:57.936909 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:57:57.937032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260810 14:57:57.937739 32579 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35345
I20260810 14:57:57.937829 32579 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:57.938020 32579 heartbeater.cc:511] Master 127.28.193.254:35345 requested a full tablet report, sending...
I20260810 14:57:57.938623 32243 ts_manager.cc:194] Registered new tserver with Master: 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669)
I20260810 14:57:57.939370 32243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:54005
I20260810 14:57:57.944196 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:57.938894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:59999 (local address 127.28.193.254:35345)
0810 14:57:57.938940 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:57.938942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:57.938972 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:57:57.939035 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:57.939036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:57.939045 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:57.939089 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:57.939391 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.940067 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.940822 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.941019 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.941082 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:57.942051 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:57.942053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:57.942053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:57.942059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:57.943247 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.943615 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.943616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.943617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.943675 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:57.943816 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:57.943817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:57.943817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:57.943920 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:57.943967 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:57.944068 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 14:57:57.944136 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:57:57.944197 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:57.938775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.193:59999)
0810 14:57:57.938911 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:57.938921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:57.938986 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:57.939092 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:57.939092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:57.939266 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:57.939316 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:57.939318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.940186 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.940187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:57.940708 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:57.940709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:57.941125 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:57.941127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:57.941194 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:57.941633 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:57.941638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:57.941974 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:57.943359 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:57.943360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:57.943360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:57.943534 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:57.943730 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:57.943730 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:57.943731 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:57.943778 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:57.943961 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:57.943962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:57.944000 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:57:57.944045 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 14:57:57.944129 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":69,"threads_started":1}
I20260810 14:57:57.944892 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32361
I20260810 14:57:57.944998 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:57:57.945168 29447 external_mini_cluster.cc:1428] Setting key 29d4b2eef1b1c2b28150ef1ef0e49c54
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:57:57.968793 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35345
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:58.049669 32589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:58.049916 32589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:58.049939 32589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:58.051084 32589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:58.051131 32589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:58.051151 32589 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:57:58.051162 32589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:58.051177 32589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:57:58.052642 32589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:58.052870 32589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:58.053088 32589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:58.057968 32601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:58.057976 32605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:58.058027 32602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:58.058136 32589 server_base.cc:1034] running on GCE node
I20260810 14:57:58.058357 32589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:58.058602 32589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:58.059773 32589 hybrid_clock.cc:648] HybridClock initialized: now 1786373878059757 us; error 29 us; skew 500 ppm
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:58.062206 32589 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:57:58.062659 32589 webserver.cc:460] Webserver started at http://127.28.193.194:33229/ using document root <none> and password file <none>
I20260810 14:57:58.062857 32589 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:58.062914 32589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:58.063043 32589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:58.064078 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "8b4ddf4bc0ff4f5bbd1def8215722855"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "8de3007f681f16fa9a8d0eda1fd16362"
server_key_iv: "c0aa1b7321b33c43a7ce05c41498d9bf"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.064416 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "8b4ddf4bc0ff4f5bbd1def8215722855"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "8de3007f681f16fa9a8d0eda1fd16362"
server_key_iv: "c0aa1b7321b33c43a7ce05c41498d9bf"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.065626 32589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:57:58.066558 32615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.066787 32589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:58.066924 32589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "8b4ddf4bc0ff4f5bbd1def8215722855"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "8de3007f681f16fa9a8d0eda1fd16362"
server_key_iv: "c0aa1b7321b33c43a7ce05c41498d9bf"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.067010 32589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:58.079821 32589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:58.081414 32589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:58.081537 32589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:58.082079 32589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:58.082121 32589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.082149 32589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:58.082166 32589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:58.089470 32589 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:42673
I20260810 14:57:58.089622 315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:42673 every 8 connection(s)
I20260810 14:57:58.089888 32589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:57:58.091625 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.082449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:42091 (local address 127.28.193.254:35345)
0810 14:57:58.082514 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.082516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.082739 (+ 223us) server_negotiation.cc:407] Connection header received
0810 14:57:58.083388 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.083389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.083399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.083462 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.084282 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.084772 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.086221 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.086428 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.086481 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.088829 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.088831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.088831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.088839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.090386 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.090750 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.090752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.090752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.090813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.091036 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.091036 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.091037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.091151 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.091206 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.091456 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.091537 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:57:58.091787 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.082425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.194:42091)
0810 14:57:58.082596 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.082646 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.083269 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.083550 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.083555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.083898 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:58.084180 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.084188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.084857 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.084858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.086137 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.086140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.086514 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.086515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.086663 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.087179 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:58.087190 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:58.088699 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:58.090473 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.090478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.090480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.090686 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.090866 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.090867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.090867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.090979 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.091205 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:58.091211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:58.091340 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.091427 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.091584 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260810 14:57:58.092211 316 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35345
I20260810 14:57:58.092288 316 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:58.092459 316 heartbeater.cc:511] Master 127.28.193.254:35345 requested a full tablet report, sending...
I20260810 14:57:58.092852 32246 ts_manager.cc:194] Registered new tserver with Master: 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673)
I20260810 14:57:58.093266 32246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:42091
I20260810 14:57:58.094010 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32589
I20260810 14:57:58.094108 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:57:58.094277 29447 external_mini_cluster.cc:1428] Setting key a7c92a5542353cd0b0a724f035fb4948
I20260810 14:57:58.098658 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.093244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:38379 (local address 127.28.193.254:35345)
0810 14:57:58.093288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.093290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.093363 (+ 73us) server_negotiation.cc:407] Connection header received
0810 14:57:58.093435 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.093436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.093454 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.093498 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.093856 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.094532 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.095264 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.095514 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.095581 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.096325 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.096326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.096327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.096334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.097583 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.097957 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.097958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.097959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.098019 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.098210 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.098210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.098211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.098342 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.098404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.098547 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.098603 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 14:57:58.098717 325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.093174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.194:38379)
0810 14:57:58.093295 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.093306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.093388 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.093515 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.093515 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.093697 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:58.093773 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.093775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.094627 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.094628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.095190 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.095193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.095579 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.095580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.095631 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.095991 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:58.095995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:58.096244 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:58.097652 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.097653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.097654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.097865 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.098085 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.098086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.098086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.098152 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.098399 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:58.098399 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:58.098448 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.098527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.098626 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:57:58.118618 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35345
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:57:58.203496 329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:57:58.203683 329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:57:58.203706 329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:57:58.204859 329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:57:58.204902 329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:57:58.204921 329 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:57:58.204933 329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:57:58.204972 329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:57:58.206449 329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:57:58.206669 329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:57:58.206872 329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:57:58.211625 337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:57:58.211627 342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:58.211732 329 server_base.cc:1034] running on GCE node
W20260810 14:57:58.211859 339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:57:58.212055 329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:57:58.212292 329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:57:58.213474 329 hybrid_clock.cc:648] HybridClock initialized: now 1786373878213446 us; error 37 us; skew 500 ppm
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:57:58.215926 329 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:57:58.216344 329 webserver.cc:460] Webserver started at http://127.28.193.195:34033/ using document root <none> and password file <none>
I20260810 14:57:58.216562 329 fs_manager.cc:359] Metadata directory not provided
I20260810 14:57:58.216600 329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:57:58.216689 329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:57:58.217630 329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "92cf383e4666211531d1e77fbb1f7d85"
server_key_iv: "186512d6362d6d2e501e76b86ae1cb9c"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.217978 329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "92cf383e4666211531d1e77fbb1f7d85"
server_key_iv: "186512d6362d6d2e501e76b86ae1cb9c"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.219151 329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:58.219769 350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.219928 329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.220008 329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
format_stamp: "Formatted at 2026-08-10 14:57:58 on dist-test-slave-cvwc"
server_key: "92cf383e4666211531d1e77fbb1f7d85"
server_key_iv: "186512d6362d6d2e501e76b86ae1cb9c"
server_key_version: "encryptionkey@0"
I20260810 14:57:58.220064 329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:57:58.230630 329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:57:58.232508 329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:57:58.232621 329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:57:58.233091 329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:57:58.233136 329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.233157 329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:57:58.233167 329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.239527 329 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35635
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:58.239874 329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:57:58.240207 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35635 every 8 connection(s)
I20260810 14:57:58.242561 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.233578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:41791 (local address 127.28.193.254:35345)
0810 14:57:58.233646 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.233649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.233829 (+ 180us) server_negotiation.cc:407] Connection header received
0810 14:57:58.234293 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.234295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.234308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.234381 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.235137 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.235869 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.237305 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.237591 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.237668 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.240148 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.240150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.240150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.240158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.241313 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.241693 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.241694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.241694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.241773 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.241977 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.241978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.241978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.242079 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.242130 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.242431 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.242499 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:57:58.242695 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.233519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.195:41791)
0810 14:57:58.233680 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.233749 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.234208 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.234453 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.234457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.234763 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:58.235036 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.235043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.235964 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.235965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.237214 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.237217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.237677 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.237678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.237851 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.238331 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:58.238341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:58.240063 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:58.241398 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.241404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.241406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.241627 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.241832 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.241833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.241833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.241930 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.242164 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:58.242179 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:58.242298 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.242394 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.242515 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260810 14:57:58.243132 531 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35345
I20260810 14:57:58.243213 531 heartbeater.cc:464] Registering TS with master...
I20260810 14:57:58.243405 531 heartbeater.cc:511] Master 127.28.193.254:35345 requested a full tablet report, sending...
I20260810 14:57:58.243829 32246 ts_manager.cc:194] Registered new tserver with Master: b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.244225 32246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:41791
I20260810 14:57:58.244843 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 329
I20260810 14:57:58.244941 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:57:58.245146 29447 external_mini_cluster.cc:1428] Setting key b8e512146c4c0b3f1bfbcd55913557af
I20260810 14:57:58.246001 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:57:58.249614 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.244250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56801 (local address 127.28.193.254:35345)
0810 14:57:58.244295 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.244296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.244352 (+ 56us) server_negotiation.cc:407] Connection header received
0810 14:57:58.244416 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.244417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.244425 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.244470 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.244827 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.245484 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.246424 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.246601 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.246654 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.247513 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.247514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.247514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.247520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.248668 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.249006 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.249006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.249006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.249056 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.249225 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.249225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.249225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.249307 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.249351 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.249510 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.249560 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:57:58.249670 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.244169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.28.193.195:56801)
0810 14:57:58.244291 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.244303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.244363 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.244486 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.244487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.244661 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:58.244732 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.244734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.245591 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.245592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.246339 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.246341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.246671 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.246672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.246747 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.247082 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:58.247087 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:58.247439 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:58.248744 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.248745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.248746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.248942 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.249112 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.249113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.249113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.249173 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.249360 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:58.249361 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:58.249420 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.249493 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.249583 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:58.269205 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.262655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.28.193.254:35345)
0810 14:57:58.262703 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.262704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.262735 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:57:58.262761 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.262762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.262770 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.262810 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.263346 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.263861 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.264586 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.264803 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.264866 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.266945 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.266947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.266947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.266954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.268219 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.268521 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.268522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.268522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.268578 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.268744 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.268745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.268746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.268856 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.268949 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.269041 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.269116 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:57:58.270655 32246 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:57:58.272194 32246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:58.278187 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.28.193.195:35635)
0810 14:57:58.274544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.274548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.274650 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:57:58.274697 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.274699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.274715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.274804 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.274997 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.276069 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277252 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.277803 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277871 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.277942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.278059 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.278098 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260810 14:57:58.278169 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35635 (local address 127.0.0.1:52282)
0810 14:57:58.274584 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.274593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.274638 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.274826 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.274829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.274831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.274898 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.274900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.276153 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.276154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.277175 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.277177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.277878 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.277959 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.277973 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.278039 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.278093 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260810 14:57:58.278363 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.28.193.193:41669)
0810 14:57:58.274673 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.274676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.274686 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:58.274719 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.274721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.274734 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.274804 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.275024 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.275746 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277094 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.277826 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277892 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.277945 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.278210 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.278253 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260810 14:57:58.278388 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.0.0.1:54778)
0810 14:57:58.274384 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.274435 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.274497 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.274874 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.274876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.274878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.274934 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.274937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.275834 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.275835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.276998 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.277000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.277982 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.278047 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.278051 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.278099 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.278141 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260810 14:57:58.278903 430 tablet_service.cc:1467] Processing CreateTablet for tablet 354b578a77334a558c5eb488269a7e52 (DEFAULT_TABLE table=test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.278968 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 354b578a77334a558c5eb488269a7e52 (DEFAULT_TABLE table=test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.279208 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b578a77334a558c5eb488269a7e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.279297 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b578a77334a558c5eb488269a7e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.281706 566 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Bootstrap starting.
I20260810 14:57:58.282420 567 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Bootstrap starting.
I20260810 14:57:58.282652 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.28.193.194:42673)
0810 14:57:58.274977 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.274982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.275121 (+ 139us) server_negotiation.cc:407] Connection header received
0810 14:57:58.275199 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.275202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.275216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.275285 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.275464 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.276378 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.277376 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.278103 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.278158 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.278206 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.282451 (+ 4245us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.282562 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":68,"threads_started":1}
I20260810 14:57:58.282267 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.274222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42673 (local address 127.0.0.1:53034)
0810 14:57:58.275021 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.275031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.275076 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.275350 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.275351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.275353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.275394 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.275396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.276450 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.276451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.277293 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.277295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.282017 (+ 4722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.282018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.282083 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.282089 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.282146 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.282192 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":60,"threads_started":1}
I20260810 14:57:58.282905 566 tablet_bootstrap.cc:654] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.283049 567 tablet_bootstrap.cc:654] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.283234 566 log.cc:824] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:58.283371 567 log.cc:824] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:58.283448 32693 tablet_service.cc:1467] Processing CreateTablet for tablet 354b578a77334a558c5eb488269a7e52 (DEFAULT_TABLE table=test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.283793 32693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 354b578a77334a558c5eb488269a7e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.283969 566 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: No bootstrap required, opened a new log
I20260810 14:57:58.284032 566 ts_tablet_manager.cc:1397] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:58.284196 567 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: No bootstrap required, opened a new log
I20260810 14:57:58.284267 567 ts_tablet_manager.cc:1397] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:58.285430 566 raft_consensus.cc:348] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.285609 566 raft_consensus.cc:374] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.285648 566 raft_consensus.cc:729] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3112e88cbbd4cd8afd1409c458fefc0, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.285782 566 consensus_queue.cc:260] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.286036 566 ts_tablet_manager.cc:1428] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:58.286181 531 heartbeater.cc:503] Master 127.28.193.254:35345 was elected leader, sending a full tablet report...
I20260810 14:57:58.286302 567 raft_consensus.cc:348] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.286465 567 raft_consensus.cc:374] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.286499 567 raft_consensus.cc:729] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59298cad414adb9877300d39aff455, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.286635 567 consensus_queue.cc:260] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.286914 567 ts_tablet_manager.cc:1428] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:57:58.287040 32579 heartbeater.cc:503] Master 127.28.193.254:35345 was elected leader, sending a full tablet report...
I20260810 14:57:58.287397 573 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Bootstrap starting.
I20260810 14:57:58.287951 573 tablet_bootstrap.cc:654] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.288218 573 log.cc:824] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Log is configured to *not* fsync() on all Append() calls
I20260810 14:57:58.288827 573 tablet_bootstrap.cc:492] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: No bootstrap required, opened a new log
I20260810 14:57:58.288878 573 ts_tablet_manager.cc:1397] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:58.290390 573 raft_consensus.cc:348] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.290517 573 raft_consensus.cc:374] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.290539 573 raft_consensus.cc:729] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.290645 573 consensus_queue.cc:260] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.290869 573 ts_tablet_manager.cc:1428] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:57:58.290920 316 heartbeater.cc:503] Master 127.28.193.254:35345 was elected leader, sending a full tablet report...
W20260810 14:57:58.340489 318 tablet.cc:2369] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:57:58.416383 571 raft_consensus.cc:482] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:58.416533 571 raft_consensus.cc:504] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.416882 571 leader_election.cc:288] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.419783 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.417271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:34341 (local address 127.28.193.195:35635)
0810 14:57:58.417324 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.417326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.417336 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:57:58.417372 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.417372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.417383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.417449 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.417623 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.418183 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419131 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.419481 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419544 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.419554 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.419681 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.419715 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:57:58.419811 32403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.417111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35635 (local address 127.28.193.193:34341)
0810 14:57:58.417168 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.417181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.417247 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.417468 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.417469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.417471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.417547 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.417548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.418261 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.418262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.419045 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.419047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.419550 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.419621 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.419625 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.419670 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.419708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":31}
I20260810 14:57:58.419802 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.417194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:40117 (local address 127.28.193.194:42673)
0810 14:57:58.417254 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.417256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.417359 (+ 103us) server_negotiation.cc:407] Connection header received
0810 14:57:58.417457 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.417459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.417468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.417520 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.417755 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.418226 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419135 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.419504 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419566 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.419575 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.419698 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.419733 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:57:58.419827 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.417111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42673 (local address 127.28.193.193:40117)
0810 14:57:58.417303 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.417311 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.417382 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.417537 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.417538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.417541 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.417626 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.417628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.418304 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.418305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.419053 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.419055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.419576 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.419577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.419640 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.419643 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.419687 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.419726 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":54,"thread_start_us":58,"threads_started":1}
I20260810 14:57:58.420133 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b578a77334a558c5eb488269a7e52" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" is_pre_election: true
I20260810 14:57:58.420148 460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b578a77334a558c5eb488269a7e52" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" is_pre_election: true
I20260810 14:57:58.420297 460 raft_consensus.cc:2454] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 0.
I20260810 14:57:58.420297 32721 raft_consensus.cc:2454] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 0.
I20260810 14:57:58.420522 32389 leader_election.cc:302] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, 8b4ddf4bc0ff4f5bbd1def8215722855; no voters:
I20260810 14:57:58.420691 571 raft_consensus.cc:2781] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:58.420742 571 raft_consensus.cc:482] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:58.420756 571 raft_consensus.cc:3037] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.421439 571 raft_consensus.cc:504] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.421561 571 leader_election.cc:288] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 election: Requested vote from peers 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.421756 460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b578a77334a558c5eb488269a7e52" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
I20260810 14:57:58.421751 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "354b578a77334a558c5eb488269a7e52" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855"
I20260810 14:57:58.421835 460 raft_consensus.cc:3037] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.421835 32721 raft_consensus.cc:3037] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.422719 460 raft_consensus.cc:2454] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 1.
I20260810 14:57:58.422719 32721 raft_consensus.cc:2454] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 1.
I20260810 14:57:58.422878 32389 leader_election.cc:302] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, 8b4ddf4bc0ff4f5bbd1def8215722855; no voters:
I20260810 14:57:58.422986 571 raft_consensus.cc:2781] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:58.423130 571 raft_consensus.cc:686] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 1 LEADER]: Becoming Leader. State: Replica: 7f59298cad414adb9877300d39aff455, State: Running, Role: LEADER
I20260810 14:57:58.423210 571 consensus_queue.cc:237] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.424057 32246 catalog_manager.cc:5495] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f59298cad414adb9877300d39aff455 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } health_report { overall_health: UNKNOWN } } }
W20260810 14:57:58.435431 32580 tablet.cc:2369] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:57:58 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373877, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:57:58.476873 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.470477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35345 (local address 127.0.0.1:58790)
0810 14:57:58.470640 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.470649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.470723 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.470851 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.470852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.471220 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:57:58.471272 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.471275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.471906 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.471907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.472435 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.472436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.472743 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.472744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.472792 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.473282 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:57:58.473286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:57:58.474683 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:57:58.475977 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.475985 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.475988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.476192 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.476343 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:57:58.476343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:57:58.476343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:57:58.476403 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:57:58.476583 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:57:58.476587 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:57:58.476629 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.476678 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.476773 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260810 14:57:58.476923 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.470521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.28.193.254:35345)
0810 14:57:58.470577 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.470580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.470712 (+ 132us) server_negotiation.cc:407] Connection header received
0810 14:57:58.470774 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.470775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.470786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.470838 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:57:58.471348 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.471835 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.472506 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.472685 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.472739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.474781 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:57:58.474783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:57:58.474783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:57:58.474790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:57:58.475902 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.476255 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.476256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.476256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.476301 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:57:58.476447 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:57:58.476448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:57:58.476448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:57:58.476542 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:57:58.476588 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.476700 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.476814 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":27}
I20260810 14:57:58.478770 32246 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:58790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:57:58.478961 32246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:58.480324 32693 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2f2bd37c4d46bc01899b617e5ad1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a7b7c368b3407b9b3c94e399a77292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:58.480330 32473 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2f2bd37c4d46bc01899b617e5ad1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a7b7c368b3407b9b3c94e399a77292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:58.480358 430 tablet_service.cc:1467] Processing CreateTablet for tablet afeb2f2bd37c4d46bc01899b617e5ad1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=73a7b7c368b3407b9b3c94e399a77292]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:57:58.480507 32693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2f2bd37c4d46bc01899b617e5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.480507 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2f2bd37c4d46bc01899b617e5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.480511 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afeb2f2bd37c4d46bc01899b617e5ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.481643 566 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: Bootstrap starting.
I20260810 14:57:58.482033 573 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: Bootstrap starting.
I20260810 14:57:58.482194 566 tablet_bootstrap.cc:654] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.482591 567 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: Bootstrap starting.
I20260810 14:57:58.482759 573 tablet_bootstrap.cc:654] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.482860 566 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: No bootstrap required, opened a new log
I20260810 14:57:58.482895 566 ts_tablet_manager.cc:1397] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:58.483024 566 raft_consensus.cc:348] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.483088 566 raft_consensus.cc:374] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.483104 566 raft_consensus.cc:729] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3112e88cbbd4cd8afd1409c458fefc0, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.483147 566 consensus_queue.cc:260] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.483235 566 ts_tablet_manager.cc:1428] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.483248 567 tablet_bootstrap.cc:654] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.483397 570 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.483493 570 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.483505 573 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: No bootstrap required, opened a new log
I20260810 14:57:58.483548 573 ts_tablet_manager.cc:1397] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:57:58.483695 573 raft_consensus.cc:348] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.483767 573 raft_consensus.cc:374] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.483783 573 raft_consensus.cc:729] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.483819 573 consensus_queue.cc:260] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.483904 573 ts_tablet_manager.cc:1428] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 14:57:58.484021 567 tablet_bootstrap.cc:492] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: No bootstrap required, opened a new log
I20260810 14:57:58.484067 567 ts_tablet_manager.cc:1397] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:57:58.484062 575 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.484148 575 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.484210 567 raft_consensus.cc:348] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.484277 567 raft_consensus.cc:374] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.484303 567 raft_consensus.cc:729] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59298cad414adb9877300d39aff455, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.484352 567 consensus_queue.cc:260] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.484438 567 ts_tablet_manager.cc:1428] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.484503 571 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.484594 571 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 14:57:58.490693 537 tablet.cc:2369] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:57:58.570703 571 raft_consensus.cc:482] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:58.570811 571 raft_consensus.cc:504] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.570968 571 leader_election.cc:288] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.571255 460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeb2f2bd37c4d46bc01899b617e5ad1" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" is_pre_election: true
I20260810 14:57:58.571260 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeb2f2bd37c4d46bc01899b617e5ad1" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" is_pre_election: true
I20260810 14:57:58.571353 460 raft_consensus.cc:2454] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 0.
I20260810 14:57:58.571367 32721 raft_consensus.cc:2454] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 0.
I20260810 14:57:58.571532 32390 leader_election.cc:302] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, b3112e88cbbd4cd8afd1409c458fefc0; no voters:
I20260810 14:57:58.571655 571 raft_consensus.cc:2781] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:58.571707 571 raft_consensus.cc:482] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:58.571729 571 raft_consensus.cc:3037] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.572464 571 raft_consensus.cc:504] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.572587 571 leader_election.cc:288] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 election: Requested vote from peers 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.572763 460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeb2f2bd37c4d46bc01899b617e5ad1" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
I20260810 14:57:58.572837 460 raft_consensus.cc:3037] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.572829 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeb2f2bd37c4d46bc01899b617e5ad1" candidate_uuid: "7f59298cad414adb9877300d39aff455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855"
I20260810 14:57:58.572919 32721 raft_consensus.cc:3037] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.573479 460 raft_consensus.cc:2454] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 1.
I20260810 14:57:58.573561 32721 raft_consensus.cc:2454] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f59298cad414adb9877300d39aff455 in term 1.
I20260810 14:57:58.573643 32390 leader_election.cc:302] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, b3112e88cbbd4cd8afd1409c458fefc0; no voters:
I20260810 14:57:58.573745 571 raft_consensus.cc:2781] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:58.573797 571 raft_consensus.cc:686] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [term 1 LEADER]: Becoming Leader. State: Replica: 7f59298cad414adb9877300d39aff455, State: Running, Role: LEADER
I20260810 14:57:58.573846 571 consensus_queue.cc:237] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.574144 580 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: TxnStatusTablet state changed. Reason: New leader 7f59298cad414adb9877300d39aff455. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.574223 580 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:58.574425 597 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:57:58.574546 597 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:57:58.574460 32243 catalog_manager.cc:5495] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f59298cad414adb9877300d39aff455 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } health_report { overall_health: UNKNOWN } } }
I20260810 14:57:58.600317 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.597622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.28.193.193:41669)
0810 14:57:58.597684 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.597685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.597693 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:57:58.597752 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.597753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.597773 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.597831 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.597984 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.598461 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.599577 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.599765 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.599827 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.600003 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.600103 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.600182 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.600225 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.600250 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:57:58.600346 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.597348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.0.0.1:54782)
0810 14:57:58.597436 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.597457 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.597519 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.597861 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.597862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.597864 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:58.597920 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.597923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.598536 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.598537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.599496 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.599500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.599835 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.599836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.599903 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.599921 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.600166 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.600173 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.600216 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.600264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:57:58.602385 460 raft_consensus.cc:1261] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Refusing update from remote peer 7f59298cad414adb9877300d39aff455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.602385 32721 raft_consensus.cc:1261] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Refusing update from remote peer 7f59298cad414adb9877300d39aff455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.602725 571 consensus_queue.cc:1035] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.602911 571 consensus_queue.cc:1035] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.604015 570 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: TxnStatusTablet state changed. Reason: New leader 7f59298cad414adb9877300d39aff455. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.604046 575 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: TxnStatusTablet state changed. Reason: New leader 7f59298cad414adb9877300d39aff455. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.604133 575 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.604135 570 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.605151 580 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.605234 580 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:58.605283 580 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.605327 580 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 7f59298cad414adb9877300d39aff455: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:57:58.605674 570 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.605775 570 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P b3112e88cbbd4cd8afd1409c458fefc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.605896 605 mvcc.cc:205] Tried to move back new op lower bound from 7316987406752673792 to 7316987406638915584. Current Snapshot: MvccSnapshot[applied={T|T < 7316987406752673792}]
I20260810 14:57:58.606527 575 tablet_replica.cc:439] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f59298cad414adb9877300d39aff455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } } }
I20260810 14:57:58.606612 575 tablet_replica.cc:442] T afeb2f2bd37c4d46bc01899b617e5ad1 P 8b4ddf4bc0ff4f5bbd1def8215722855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:57:58.607621 606 mvcc.cc:205] Tried to move back new op lower bound from 7316987406752673792 to 7316987406638915584. Current Snapshot: MvccSnapshot[applied={T|T < 7316987406752673792}]
I20260810 14:57:58.609740 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.607478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.28.193.193:41669)
0810 14:57:58.607576 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.607577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.607582 (+ 5us) server_negotiation.cc:407] Connection header received
0810 14:57:58.607598 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.607598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.607607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.607646 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.607856 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.608351 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.609118 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.609322 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.609375 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.609513 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.609546 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.609587 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.609637 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.609656 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 14:57:58.614410 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.612145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38725 (local address 127.28.193.193:41669)
0810 14:57:58.612183 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.612184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.612315 (+ 131us) server_negotiation.cc:407] Connection header received
0810 14:57:58.612357 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.612358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.612366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.612411 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.612537 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.613014 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.613773 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.613950 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.614004 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.614178 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.614206 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.614254 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.614306 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.614322 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
I20260810 14:57:58.614427 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.612088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.28.193.193:38725)
0810 14:57:58.612257 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.612265 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.612316 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.612417 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.612418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.612419 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:58.612477 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.612479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.613083 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.613084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.613669 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.613671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.614007 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.614008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.614082 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.614098 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.614254 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.614256 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.614302 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.614341 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260810 14:57:58.616405 460 raft_consensus.cc:1261] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Refusing update from remote peer 7f59298cad414adb9877300d39aff455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.616439 32721 raft_consensus.cc:1261] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Refusing update from remote peer 7f59298cad414adb9877300d39aff455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.616612 621 consensus_queue.cc:1035] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.616746 580 consensus_queue.cc:1035] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.618306 603 mvcc.cc:205] Tried to move back new op lower bound from 7316987406811459584 to 7316987406022045696. Current Snapshot: MvccSnapshot[applied={T|T < 7316987406811459584}]
I20260810 14:57:58.628539 32243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:58.628659 32243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:57:58.629412 32243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Sending DeleteTablet for 3 replicas of tablet 354b578a77334a558c5eb488269a7e52
I20260810 14:57:58.629747 32472 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b578a77334a558c5eb488269a7e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:58 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260810 14:57:58.629765 32693 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b578a77334a558c5eb488269a7e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:58 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:53034
I20260810 14:57:58.629845 430 tablet_service.cc:1514] Processing DeleteTablet for tablet 354b578a77334a558c5eb488269a7e52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:57:58 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260810 14:57:58.630053 630 tablet_replica.cc:330] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: stopping tablet replica
I20260810 14:57:58.630174 630 raft_consensus.cc:2229] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:57:58.630321 627 tablet_replica.cc:330] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: stopping tablet replica
I20260810 14:57:58.630311 32243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:57:58.630427 627 raft_consensus.cc:2229] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 14:57:58.630445 32243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:57:58.630475 627 raft_consensus.cc:2258] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:58.630576 630 raft_consensus.cc:2258] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:58.630934 628 tablet_replica.cc:330] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: stopping tablet replica
I20260810 14:57:58.631034 628 raft_consensus.cc:2229] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:57:58.631261 628 raft_consensus.cc:2258] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:57:58.631692 32472 tablet_service.cc:1467] Processing CreateTablet for tablet 50a47a18714d4e61a374616dbfab7b52 (DEFAULT_TABLE table=test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.631767 430 tablet_service.cc:1467] Processing CreateTablet for tablet 50a47a18714d4e61a374616dbfab7b52 (DEFAULT_TABLE table=test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.631819 627 ts_tablet_manager.cc:1905] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:58.631829 32472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a47a18714d4e61a374616dbfab7b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.631887 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a47a18714d4e61a374616dbfab7b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.633174 567 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Bootstrap starting.
I20260810 14:57:58.633275 566 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Bootstrap starting.
I20260810 14:57:58.633584 567 tablet_bootstrap.cc:654] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.633628 566 tablet_bootstrap.cc:654] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.634150 628 ts_tablet_manager.cc:1905] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:58.634173 32693 tablet_service.cc:1467] Processing CreateTablet for tablet 50a47a18714d4e61a374616dbfab7b52 (DEFAULT_TABLE table=test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:57:58.634228 567 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: No bootstrap required, opened a new log
I20260810 14:57:58.634272 567 ts_tablet_manager.cc:1397] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:58.634399 567 raft_consensus.cc:348] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.634470 567 raft_consensus.cc:374] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.634488 567 raft_consensus.cc:729] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59298cad414adb9877300d39aff455, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.634528 567 consensus_queue.cc:260] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.634528 630 ts_tablet_manager.cc:1905] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:57:58.634613 567 ts_tablet_manager.cc:1428] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.636171 32693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a47a18714d4e61a374616dbfab7b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:57:58.636287 630 ts_tablet_manager.cc:1918] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:58.636385 566 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: No bootstrap required, opened a new log
I20260810 14:57:58.636485 566 ts_tablet_manager.cc:1397] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 14:57:58.636613 566 raft_consensus.cc:348] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.636677 566 raft_consensus.cc:374] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.636693 566 raft_consensus.cc:729] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3112e88cbbd4cd8afd1409c458fefc0, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.636731 566 consensus_queue.cc:260] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.636818 566 ts_tablet_manager.cc:1428] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.636399 630 log.cc:1196] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/354b578a77334a558c5eb488269a7e52
I20260810 14:57:58.637249 630 ts_tablet_manager.cc:1939] T 354b578a77334a558c5eb488269a7e52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting consensus metadata
I20260810 14:57:58.637524 573 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Bootstrap starting.
I20260810 14:57:58.637646 32221 catalog_manager.cc:4841] TS b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635): tablet 354b578a77334a558c5eb488269a7e52 (table test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]) successfully deleted
I20260810 14:57:58.637948 573 tablet_bootstrap.cc:654] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Neither blocks nor log segments found. Creating new log.
I20260810 14:57:58.638566 573 tablet_bootstrap.cc:492] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: No bootstrap required, opened a new log
I20260810 14:57:58.638664 573 ts_tablet_manager.cc:1397] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:57:58.638698 628 ts_tablet_manager.cc:1918] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:58.638778 628 log.cc:1196] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/354b578a77334a558c5eb488269a7e52
I20260810 14:57:58.638798 573 raft_consensus.cc:348] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.638863 573 raft_consensus.cc:374] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:57:58.638885 573 raft_consensus.cc:729] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Initialized, Role: FOLLOWER
I20260810 14:57:58.638921 573 consensus_queue.cc:260] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.639000 573 ts_tablet_manager.cc:1428] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:57:58.639029 628 ts_tablet_manager.cc:1939] T 354b578a77334a558c5eb488269a7e52 P 7f59298cad414adb9877300d39aff455: Deleting consensus metadata
I20260810 14:57:58.639406 32222 catalog_manager.cc:4841] TS 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669): tablet 354b578a77334a558c5eb488269a7e52 (table test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]) successfully deleted
I20260810 14:57:58.638578 627 ts_tablet_manager.cc:1918] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:57:58.639521 627 log.cc:1196] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/354b578a77334a558c5eb488269a7e52
I20260810 14:57:58.639755 627 ts_tablet_manager.cc:1939] T 354b578a77334a558c5eb488269a7e52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting consensus metadata
I20260810 14:57:58.640100 32220 catalog_manager.cc:4841] TS 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673): tablet 354b578a77334a558c5eb488269a7e52 (table test-table [id=322f7b52ac4a468f93bfd1b90d9cfb8f]) successfully deleted
I20260810 14:57:58.641775 575 raft_consensus.cc:482] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:57:58.641840 575 raft_consensus.cc:504] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.642143 575 leader_election.cc:288] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.645649 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.642513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:52767 (local address 127.28.193.195:35635)
0810 14:57:58.642605 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.642607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.642614 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:57:58.642635 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.642636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.642644 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.642689 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.642846 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.643594 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.644987 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.645346 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.645417 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.645427 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.645555 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.645590 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:57:58.645673 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.642296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35635 (local address 127.28.193.194:52767)
0810 14:57:58.642427 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.642440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.642490 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.642706 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.642707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.642709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.642761 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.642763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.643680 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.643681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.644874 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.644877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.645417 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.645418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.645489 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.645499 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.645543 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.645583 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260810 14:57:58.646051 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a47a18714d4e61a374616dbfab7b52" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" is_pre_election: true
I20260810 14:57:58.646136 458 raft_consensus.cc:2454] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:57:58.646296 32620 leader_election.cc:302] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4ddf4bc0ff4f5bbd1def8215722855, b3112e88cbbd4cd8afd1409c458fefc0; no voters:
I20260810 14:57:58.646420 575 raft_consensus.cc:2781] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:57:58.646452 575 raft_consensus.cc:482] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:57:58.646473 575 raft_consensus.cc:3037] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.646814 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.642563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:51437 (local address 127.28.193.193:41669)
0810 14:57:58.642762 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.642763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.642771 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:57:58.642789 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.642790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.642798 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.642844 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:57:58.643861 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.644585 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.645901 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.646451 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.646497 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.646506 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.646717 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.646754 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260810 14:57:58.647051 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.642296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.28.193.194:51437)
0810 14:57:58.642453 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.642464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.642520 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.643711 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.643712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.643715 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:57:58.643785 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.643787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.644668 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.644669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.645799 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.645803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.646563 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.646564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.646642 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.646647 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.646929 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.646973 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":86,"threads_started":1}
I20260810 14:57:58.647126 575 raft_consensus.cc:504] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.647259 575 leader_election.cc:288] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Requested vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:57:58.647280 32504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a47a18714d4e61a374616dbfab7b52" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455" is_pre_election: true
I20260810 14:57:58.647392 32504 raft_consensus.cc:2454] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:57:58.647444 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a47a18714d4e61a374616dbfab7b52" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
I20260810 14:57:58.647475 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a47a18714d4e61a374616dbfab7b52" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455"
I20260810 14:57:58.647519 458 raft_consensus.cc:3037] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.647536 32503 raft_consensus.cc:3037] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:57:58.648276 32503 raft_consensus.cc:2454] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:57:58.648471 458 raft_consensus.cc:2454] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:57:58.648466 32622 leader_election.cc:302] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, 8b4ddf4bc0ff4f5bbd1def8215722855; no voters:
I20260810 14:57:58.648550 575 raft_consensus.cc:2781] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:57:58.648697 575 raft_consensus.cc:686] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Becoming Leader. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Running, Role: LEADER
I20260810 14:57:58.648767 575 consensus_queue.cc:237] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:57:58.649418 32243 catalog_manager.cc:5495] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194). New cstate: current_term: 1 leader_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } health_report { overall_health: UNKNOWN } } }
I20260810 14:57:58.659966 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.657825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.28.193.194:42673)
0810 14:57:58.657878 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.657880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.657888 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:57:58.657921 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.657921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.657939 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.657998 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.658100 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.658581 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.659279 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.659472 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.659526 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.659668 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.659768 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.659829 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.659878 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.659897 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:57:58.665079 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.662525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:53699 (local address 127.28.193.193:41669)
0810 14:57:58.662575 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.662577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.662615 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:57:58.662676 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.662677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.662687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.662730 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.662866 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.663325 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.664434 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.664613 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.664666 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.664873 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.664905 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.664936 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.665011 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.665025 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:57:58.665203 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.662415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.28.193.194:53699)
0810 14:57:58.662560 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.662571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.662630 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.662742 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.662742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.662744 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:58.662807 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.662808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.663412 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.663413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.664335 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.664339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.664707 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.664709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.664777 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.664793 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.664955 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.664957 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.665009 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.665055 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260810 14:57:58.667639 458 raft_consensus.cc:1261] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.667716 32503 raft_consensus.cc:1261] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:57:58.667872 575 consensus_queue.cc:1035] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.668001 638 consensus_queue.cc:1035] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:57:58.675334 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.672757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42673 (local address 127.28.193.193:47093)
0810 14:57:58.672840 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 14:57:58.672850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:57:58.672908 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:57:58.673027 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:57:58.673028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:57:58.673029 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:57:58.673072 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.673074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.673939 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.673940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.674560 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:57:58.674563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.675005 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.675006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:57:58.675049 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.675055 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.675216 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.675217 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 14:57:58.675240 (+ 23us) client_negotiation.cc:240] Negotiation successful
0810 14:57:58.675274 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:57:58.675339 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:57:58.672795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:47093 (local address 127.28.193.194:42673)
0810 14:57:58.672838 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:57:58.672839 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:57:58.672896 (+ 57us) server_negotiation.cc:407] Connection header received
0810 14:57:58.672959 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:57:58.672960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:57:58.672971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:57:58.673025 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:57:58.673125 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.673857 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.674643 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:57:58.674927 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:57:58.675007 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:57:58.675126 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:57:58.675159 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:57:58.675216 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:57:58.675250 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 14:57:58.675270 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 14:57:58.676460 606 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:58:00.198869 32243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:00.198977 32243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:00.199828 32243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Sending DeleteTablet for 3 replicas of tablet 50a47a18714d4e61a374616dbfab7b52
I20260810 14:58:00.200081 32693 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a47a18714d4e61a374616dbfab7b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:53034
I20260810 14:58:00.200127 32472 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a47a18714d4e61a374616dbfab7b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260810 14:58:00.200170 430 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a47a18714d4e61a374616dbfab7b52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260810 14:58:00.200475 692 tablet_replica.cc:330] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: stopping tablet replica
I20260810 14:58:00.200575 692 raft_consensus.cc:2229] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:00.200629 694 tablet_replica.cc:330] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: stopping tablet replica
I20260810 14:58:00.200672 692 raft_consensus.cc:2258] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:00.200731 694 raft_consensus.cc:2229] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:00.200788 694 raft_consensus.cc:2258] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:00.200935 692 ts_tablet_manager.cc:1905] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.201025 694 ts_tablet_manager.cc:1905] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.202579 695 tablet_replica.cc:330] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: stopping tablet replica
I20260810 14:58:00.202633 692 ts_tablet_manager.cc:1918] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:58:00.202682 695 raft_consensus.cc:2229] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:00.202703 692 log.cc:1196] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/50a47a18714d4e61a374616dbfab7b52
I20260810 14:58:00.202757 695 raft_consensus.cc:2258] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:00.202955 692 ts_tablet_manager.cc:1939] T 50a47a18714d4e61a374616dbfab7b52 P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting consensus metadata
I20260810 14:58:00.203047 695 ts_tablet_manager.cc:1905] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.203246 32220 catalog_manager.cc:4841] TS 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673): tablet 50a47a18714d4e61a374616dbfab7b52 (table test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]) successfully deleted
I20260810 14:58:00.203675 694 ts_tablet_manager.cc:1918] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:58:00.203732 694 log.cc:1196] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/50a47a18714d4e61a374616dbfab7b52
I20260810 14:58:00.203922 694 ts_tablet_manager.cc:1939] T 50a47a18714d4e61a374616dbfab7b52 P 7f59298cad414adb9877300d39aff455: Deleting consensus metadata
I20260810 14:58:00.204259 32222 catalog_manager.cc:4841] TS 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669): tablet 50a47a18714d4e61a374616dbfab7b52 (table test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]) successfully deleted
I20260810 14:58:00.204389 695 ts_tablet_manager.cc:1918] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:58:00.204442 695 log.cc:1196] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/50a47a18714d4e61a374616dbfab7b52
I20260810 14:58:00.204627 695 ts_tablet_manager.cc:1939] T 50a47a18714d4e61a374616dbfab7b52 P b3112e88cbbd4cd8afd1409c458fefc0: Deleting consensus metadata
I20260810 14:58:00.204911 32221 catalog_manager.cc:4841] TS b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635): tablet 50a47a18714d4e61a374616dbfab7b52 (table test-table [id=5ee44a43d0eb4a14bf7a3cde846f3f7d]) successfully deleted
Aug 10 14:58:00 dist-test-slave-cvwc krb5kdc[32174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:58:00 dist-test-slave-cvwc krb5kdc[32174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:00.226032 708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:00.219131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.28.193.254:35345)
0810 14:58:00.219275 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:00.219277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:00.219287 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:00.219349 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:00.219350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:00.219361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:00.219418 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:00.219934 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:00.220448 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:00.221159 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:00.221365 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:00.221419 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:00.223692 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:00.223694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:00.223694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:00.223702 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:00.225025 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:00.225336 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:00.225337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:00.225337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:00.225387 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:00.225514 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:00.225514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:00.225514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:00.225609 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:00.225684 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:00.225890 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 14:58:00.225957 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260810 14:58:00.227320 32243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:00.227478 32243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:00.228715 32693 tablet_service.cc:1467] Processing CreateTablet for tablet eb26fe5d869143e9849da9be958b47bf (DEFAULT_TABLE table=test-table [id=a25efdb521b04438b2604b62873ac74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.228710 430 tablet_service.cc:1467] Processing CreateTablet for tablet eb26fe5d869143e9849da9be958b47bf (DEFAULT_TABLE table=test-table [id=a25efdb521b04438b2604b62873ac74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.228722 32472 tablet_service.cc:1467] Processing CreateTablet for tablet eb26fe5d869143e9849da9be958b47bf (DEFAULT_TABLE table=test-table [id=a25efdb521b04438b2604b62873ac74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.228854 32693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb26fe5d869143e9849da9be958b47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.228854 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb26fe5d869143e9849da9be958b47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.228857 32472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb26fe5d869143e9849da9be958b47bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.230070 709 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Bootstrap starting.
I20260810 14:58:00.230216 710 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Bootstrap starting.
I20260810 14:58:00.230561 709 tablet_bootstrap.cc:654] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.230635 710 tablet_bootstrap.cc:654] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.230929 711 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Bootstrap starting.
I20260810 14:58:00.231263 711 tablet_bootstrap.cc:654] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.231292 709 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: No bootstrap required, opened a new log
I20260810 14:58:00.231338 709 ts_tablet_manager.cc:1397] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:00.231458 709 raft_consensus.cc:348] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.231518 709 raft_consensus.cc:374] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.231534 709 raft_consensus.cc:729] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59298cad414adb9877300d39aff455, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.231604 709 consensus_queue.cc:260] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.231706 709 ts_tablet_manager.cc:1428] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:00.231912 710 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: No bootstrap required, opened a new log
I20260810 14:58:00.231947 711 tablet_bootstrap.cc:492] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: No bootstrap required, opened a new log
I20260810 14:58:00.231961 710 ts_tablet_manager.cc:1397] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:00.231992 711 ts_tablet_manager.cc:1397] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:00.232069 710 raft_consensus.cc:348] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.232136 710 raft_consensus.cc:374] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.232156 710 raft_consensus.cc:729] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.232143 711 raft_consensus.cc:348] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.232220 710 consensus_queue.cc:260] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.232290 711 raft_consensus.cc:374] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.232321 710 ts_tablet_manager.cc:1428] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:00.232343 711 raft_consensus.cc:729] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3112e88cbbd4cd8afd1409c458fefc0, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.232514 711 consensus_queue.cc:260] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.232712 711 ts_tablet_manager.cc:1428] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 14:58:00.334743 681 raft_consensus.cc:482] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:00.334860 681 raft_consensus.cc:504] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.335041 681 leader_election.cc:288] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:58:00.335310 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb26fe5d869143e9849da9be958b47bf" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455" is_pre_election: true
I20260810 14:58:00.335304 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb26fe5d869143e9849da9be958b47bf" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" is_pre_election: true
I20260810 14:58:00.335441 32503 raft_consensus.cc:2454] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:58:00.335441 458 raft_consensus.cc:2454] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:58:00.335631 32622 leader_election.cc:302] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, 8b4ddf4bc0ff4f5bbd1def8215722855; no voters:
I20260810 14:58:00.335745 681 raft_consensus.cc:2781] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:00.335803 681 raft_consensus.cc:482] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:00.335830 681 raft_consensus.cc:3037] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.336570 681 raft_consensus.cc:504] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.336690 681 leader_election.cc:288] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Requested vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:58:00.336892 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb26fe5d869143e9849da9be958b47bf" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
I20260810 14:58:00.336891 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb26fe5d869143e9849da9be958b47bf" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455"
I20260810 14:58:00.336975 32503 raft_consensus.cc:3037] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.336975 458 raft_consensus.cc:3037] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.337766 458 raft_consensus.cc:2454] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:58:00.337800 32503 raft_consensus.cc:2454] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:58:00.337952 32620 leader_election.cc:302] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4ddf4bc0ff4f5bbd1def8215722855, b3112e88cbbd4cd8afd1409c458fefc0; no voters:
I20260810 14:58:00.338053 681 raft_consensus.cc:2781] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:00.338104 681 raft_consensus.cc:686] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Becoming Leader. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Running, Role: LEADER
I20260810 14:58:00.338151 681 consensus_queue.cc:237] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.338835 32246 catalog_manager.cc:5495] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194). New cstate: current_term: 1 leader_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } health_report { overall_health: UNKNOWN } } }
I20260810 14:58:00.348556 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:00.346537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.28.193.194:42673)
0810 14:58:00.346689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:00.346691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:00.346697 (+ 6us) server_negotiation.cc:407] Connection header received
0810 14:58:00.346716 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:00.346716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:00.346726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:00.346776 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:00.346893 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:00.347370 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:00.348066 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:00.348245 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:00.348288 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:00.348381 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:00.348408 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:00.348437 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:00.348474 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 14:58:00.348488 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260810 14:58:00.350541 458 raft_consensus.cc:1261] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:00.350598 32503 raft_consensus.cc:1261] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:00.350736 681 consensus_queue.cc:1035] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:00.350857 677 consensus_queue.cc:1035] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:00.361021 32242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:00.361117 32242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:00.361737 32242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Sending DeleteTablet for 3 replicas of tablet eb26fe5d869143e9849da9be958b47bf
I20260810 14:58:00.361984 32472 tablet_service.cc:1514] Processing DeleteTablet for tablet eb26fe5d869143e9849da9be958b47bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260810 14:58:00.361987 32693 tablet_service.cc:1514] Processing DeleteTablet for tablet eb26fe5d869143e9849da9be958b47bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:53034
I20260810 14:58:00.362022 430 tablet_service.cc:1514] Processing DeleteTablet for tablet eb26fe5d869143e9849da9be958b47bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:00 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260810 14:58:00.362097 692 tablet_replica.cc:330] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: stopping tablet replica
I20260810 14:58:00.362110 695 tablet_replica.cc:330] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: stopping tablet replica
I20260810 14:58:00.362152 692 raft_consensus.cc:2229] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:00.362160 695 raft_consensus.cc:2229] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:00.362196 692 raft_consensus.cc:2258] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:00.362202 695 raft_consensus.cc:2258] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:00.362259 694 tablet_replica.cc:330] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: stopping tablet replica
I20260810 14:58:00.362236 32242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:58:00.362315 694 raft_consensus.cc:2229] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:00.362355 694 raft_consensus.cc:2258] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 14:58:00.362370 32242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:00.362630 695 ts_tablet_manager.cc:1905] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.362839 692 ts_tablet_manager.cc:1905] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.363588 32693 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd115cdf704cc7a4200951e2a5708d (DEFAULT_TABLE table=test-table [id=b78cbf4f362e4ba8850b543bdf66bbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.363698 32693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd115cdf704cc7a4200951e2a5708d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.364269 694 ts_tablet_manager.cc:1905] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:00.364296 695 ts_tablet_manager.cc:1918] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:00.364295 692 ts_tablet_manager.cc:1918] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:00.364346 692 log.cc:1196] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/eb26fe5d869143e9849da9be958b47bf
I20260810 14:58:00.364346 695 log.cc:1196] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/eb26fe5d869143e9849da9be958b47bf
I20260810 14:58:00.364555 692 ts_tablet_manager.cc:1939] T eb26fe5d869143e9849da9be958b47bf P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting consensus metadata
I20260810 14:58:00.364571 695 ts_tablet_manager.cc:1939] T eb26fe5d869143e9849da9be958b47bf P b3112e88cbbd4cd8afd1409c458fefc0: Deleting consensus metadata
I20260810 14:58:00.364822 710 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Bootstrap starting.
I20260810 14:58:00.364868 430 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd115cdf704cc7a4200951e2a5708d (DEFAULT_TABLE table=test-table [id=b78cbf4f362e4ba8850b543bdf66bbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.364964 430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd115cdf704cc7a4200951e2a5708d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.365078 710 tablet_bootstrap.cc:654] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.365268 32221 catalog_manager.cc:4841] TS b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635): tablet eb26fe5d869143e9849da9be958b47bf (table test-table [id=a25efdb521b04438b2604b62873ac74c]) successfully deleted
I20260810 14:58:00.365767 694 ts_tablet_manager.cc:1918] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:00.365816 694 log.cc:1196] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/eb26fe5d869143e9849da9be958b47bf
I20260810 14:58:00.366034 694 ts_tablet_manager.cc:1939] T eb26fe5d869143e9849da9be958b47bf P 7f59298cad414adb9877300d39aff455: Deleting consensus metadata
I20260810 14:58:00.366070 711 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Bootstrap starting.
I20260810 14:58:00.366185 32220 catalog_manager.cc:4841] TS 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673): tablet eb26fe5d869143e9849da9be958b47bf (table test-table [id=a25efdb521b04438b2604b62873ac74c]) successfully deleted
I20260810 14:58:00.366392 711 tablet_bootstrap.cc:654] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.366436 32222 catalog_manager.cc:4841] TS 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669): tablet eb26fe5d869143e9849da9be958b47bf (table test-table [id=a25efdb521b04438b2604b62873ac74c]) successfully deleted
I20260810 14:58:00.366735 32472 tablet_service.cc:1467] Processing CreateTablet for tablet e5cd115cdf704cc7a4200951e2a5708d (DEFAULT_TABLE table=test-table [id=b78cbf4f362e4ba8850b543bdf66bbb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:00.366835 32472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5cd115cdf704cc7a4200951e2a5708d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:00.366879 710 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: No bootstrap required, opened a new log
I20260810 14:58:00.366918 710 ts_tablet_manager.cc:1397] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:58:00.367030 710 raft_consensus.cc:348] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.367089 710 raft_consensus.cc:374] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.367122 710 raft_consensus.cc:729] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.367161 710 consensus_queue.cc:260] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.367241 710 ts_tablet_manager.cc:1428] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:00.367880 709 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455: Bootstrap starting.
I20260810 14:58:00.368078 711 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: No bootstrap required, opened a new log
I20260810 14:58:00.368124 711 ts_tablet_manager.cc:1397] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:00.368211 709 tablet_bootstrap.cc:654] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:00.368258 711 raft_consensus.cc:348] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.368319 711 raft_consensus.cc:374] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.368338 711 raft_consensus.cc:729] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3112e88cbbd4cd8afd1409c458fefc0, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.368381 711 consensus_queue.cc:260] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.368474 711 ts_tablet_manager.cc:1428] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:00.368705 709 tablet_bootstrap.cc:492] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455: No bootstrap required, opened a new log
I20260810 14:58:00.368757 709 ts_tablet_manager.cc:1397] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:00.368855 709 raft_consensus.cc:348] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.368921 709 raft_consensus.cc:374] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:00.368955 709 raft_consensus.cc:729] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f59298cad414adb9877300d39aff455, State: Initialized, Role: FOLLOWER
I20260810 14:58:00.368999 709 consensus_queue.cc:260] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.369086 709 ts_tablet_manager.cc:1428] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:00.399960 660 raft_consensus.cc:482] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:00.400058 660 raft_consensus.cc:504] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.400219 660 leader_election.cc:288] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:58:00.400403 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd115cdf704cc7a4200951e2a5708d" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" is_pre_election: true
I20260810 14:58:00.400403 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd115cdf704cc7a4200951e2a5708d" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455" is_pre_election: true
I20260810 14:58:00.400488 458 raft_consensus.cc:2454] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:58:00.400506 32503 raft_consensus.cc:2454] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 0.
I20260810 14:58:00.400624 32620 leader_election.cc:302] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b4ddf4bc0ff4f5bbd1def8215722855, b3112e88cbbd4cd8afd1409c458fefc0; no voters:
I20260810 14:58:00.400717 660 raft_consensus.cc:2781] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:00.400771 660 raft_consensus.cc:482] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:00.400790 660 raft_consensus.cc:3037] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.401479 660 raft_consensus.cc:504] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.401597 660 leader_election.cc:288] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Requested vote from peers 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669), b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635)
I20260810 14:58:00.401809 32503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd115cdf704cc7a4200951e2a5708d" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f59298cad414adb9877300d39aff455"
I20260810 14:58:00.401799 458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5cd115cdf704cc7a4200951e2a5708d" candidate_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3112e88cbbd4cd8afd1409c458fefc0"
I20260810 14:58:00.401887 32503 raft_consensus.cc:3037] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.401887 458 raft_consensus.cc:3037] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:00.402645 32503 raft_consensus.cc:2454] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:58:00.402645 458 raft_consensus.cc:2454] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b4ddf4bc0ff4f5bbd1def8215722855 in term 1.
I20260810 14:58:00.402812 32622 leader_election.cc:302] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f59298cad414adb9877300d39aff455, 8b4ddf4bc0ff4f5bbd1def8215722855; no voters:
I20260810 14:58:00.402905 660 raft_consensus.cc:2781] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:00.402951 660 raft_consensus.cc:686] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Becoming Leader. State: Replica: 8b4ddf4bc0ff4f5bbd1def8215722855, State: Running, Role: LEADER
I20260810 14:58:00.402999 660 consensus_queue.cc:237] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } }
I20260810 14:58:00.403532 32242 catalog_manager.cc:5495] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194). New cstate: current_term: 1 leader_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b4ddf4bc0ff4f5bbd1def8215722855" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 } health_report { overall_health: UNKNOWN } } }
I20260810 14:58:00.416291 32503 raft_consensus.cc:1261] T e5cd115cdf704cc7a4200951e2a5708d P 7f59298cad414adb9877300d39aff455 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:00.416342 458 raft_consensus.cc:1261] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Refusing update from remote peer 8b4ddf4bc0ff4f5bbd1def8215722855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:00.416515 660 consensus_queue.cc:1035] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3112e88cbbd4cd8afd1409c458fefc0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:00.416607 681 consensus_queue.cc:1035] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f59298cad414adb9877300d39aff455" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:58:00.420446 664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:58:01.937520 32243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:01.937639 32243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:01.938552 32243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1d54faf77b24a36ad253f01cb537c8a: Sending DeleteTablet for 3 replicas of tablet e5cd115cdf704cc7a4200951e2a5708d
I20260810 14:58:01.938832 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd115cdf704cc7a4200951e2a5708d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:01 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260810 14:58:01.938937 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32361
I20260810 14:58:01.938949 430 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd115cdf704cc7a4200951e2a5708d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:01 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:52282
I20260810 14:58:01.939275 32693 tablet_service.cc:1514] Processing DeleteTablet for tablet e5cd115cdf704cc7a4200951e2a5708d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:01 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:53034
I20260810 14:58:01.947580 780 tablet_replica.cc:330] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: stopping tablet replica
I20260810 14:58:01.947670 780 raft_consensus.cc:2229] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:01.947754 780 raft_consensus.cc:2258] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:01.948050 780 ts_tablet_manager.cc:1905] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:01.949891 778 tablet_replica.cc:330] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: stopping tablet replica
I20260810 14:58:01.949980 778 raft_consensus.cc:2229] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:01.950062 778 raft_consensus.cc:2258] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:01.950084 780 ts_tablet_manager.cc:1918] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:58:01.950304 780 log.cc:1196] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/e5cd115cdf704cc7a4200951e2a5708d
I20260810 14:58:01.950304 778 ts_tablet_manager.cc:1905] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:01.950625 780 ts_tablet_manager.cc:1939] T e5cd115cdf704cc7a4200951e2a5708d P 8b4ddf4bc0ff4f5bbd1def8215722855: Deleting consensus metadata
I20260810 14:58:01.951016 32220 catalog_manager.cc:4841] TS 8b4ddf4bc0ff4f5bbd1def8215722855 (127.28.193.194:42673): tablet e5cd115cdf704cc7a4200951e2a5708d (table test-table [id=b78cbf4f362e4ba8850b543bdf66bbb7]) successfully deleted
I20260810 14:58:01.951565 778 ts_tablet_manager.cc:1918] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:58:01.951618 778 log.cc:1196] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/e5cd115cdf704cc7a4200951e2a5708d
I20260810 14:58:01.951790 778 ts_tablet_manager.cc:1939] T e5cd115cdf704cc7a4200951e2a5708d P b3112e88cbbd4cd8afd1409c458fefc0: Deleting consensus metadata
I20260810 14:58:01.952080 32221 catalog_manager.cc:4841] TS b3112e88cbbd4cd8afd1409c458fefc0 (127.28.193.195:35635): tablet e5cd115cdf704cc7a4200951e2a5708d (table test-table [id=b78cbf4f362e4ba8850b543bdf66bbb7]) successfully deleted
W20260810 14:58:01.955056 32222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:41669 (error 108)
I20260810 14:58:01.955380 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32589
I20260810 14:58:01.962821 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:01.955452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:41669 (local address 127.0.0.1:54800)
0810 14:58:01.962673 (+ 7221us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:01.962767 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:41669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7183,"thread_start_us":69,"threads_started":1}
W20260810 14:58:01.962949 32222 catalog_manager.cc:4584] TS 7f59298cad414adb9877300d39aff455 (127.28.193.193:41669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5cd115cdf704cc7a4200951e2a5708d: Network error: Client connection negotiation failed: client connection to 127.28.193.193:41669: connect: Connection refused (error 111)
I20260810 14:58:01.972138 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 329
I20260810 14:58:01.980997 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32194
2026-08-10T14:58:01Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5648 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:02 dist-test-slave-cvwc krb5kdc[787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:02 dist-test-slave-cvwc krb5kdc[787](info): set up 2 sockets
Aug 10 14:58:02 dist-test-slave-cvwc krb5kdc[787](info): commencing operation
krb5kdc: starting...
W20260810 14:58:03.142208 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:03.260438 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:40639
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:03.340816 803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:03.341010 803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:03.341032 803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:03.342123 803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:03.342155 803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:03.342167 803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:03.342178 803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:03.342190 803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:03.343601 803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:03.343801 803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:03.343972 803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:03.348783 814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.348812 803 server_base.cc:1034] running on GCE node
W20260810 14:58:03.348784 816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.348814 812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.349244 803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:03.349484 803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:03.350631 803 hybrid_clock.cc:648] HybridClock initialized: now 1786373883350613 us; error 34 us; skew 500 ppm
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:03.353042 803 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:03.353466 803 webserver.cc:460] Webserver started at http://127.28.193.254:38073/ using document root <none> and password file <none>
I20260810 14:58:03.353690 803 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:03.353768 803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:03.353895 803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:03.354913 803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "e40f0c5fc0b346bd984b23454280496a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "066c1c59c7b0da044a25553d9ad98bd9"
server_key_iv: "9917b766276ca7862757cb026c67acbb"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.355258 803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "e40f0c5fc0b346bd984b23454280496a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "066c1c59c7b0da044a25553d9ad98bd9"
server_key_iv: "9917b766276ca7862757cb026c67acbb"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.356520 803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:03.357170 832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.357314 803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:03.357406 803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "e40f0c5fc0b346bd984b23454280496a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "066c1c59c7b0da044a25553d9ad98bd9"
server_key_iv: "9917b766276ca7862757cb026c67acbb"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.357467 803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:03.387240 803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:03.389374 803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:03.389521 803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:03.393244 803 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:40639
I20260810 14:58:03.393291 913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:40639 every 8 connection(s)
I20260810 14:58:03.393623 803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:03.394543 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:03.395331 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 803
I20260810 14:58:03.395406 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:03.395572 29447 external_mini_cluster.cc:1428] Setting key 2c463673ed9af02e600f7f17b0f3a1f3
I20260810 14:58:03.397317 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: Bootstrap starting.
I20260810 14:58:03.397905 914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:03.398168 914 log.cc:824] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:03.398815 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: No bootstrap required, opened a new log
I20260810 14:58:03.400619 914 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } }
I20260810 14:58:03.400753 914 raft_consensus.cc:374] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:03.400787 914 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40f0c5fc0b346bd984b23454280496a, State: Initialized, Role: FOLLOWER
I20260810 14:58:03.400923 914 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } }
I20260810 14:58:03.401010 914 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:03.401041 914 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:03.401091 914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 0 FOLLOWER]: Advancing to term 1
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:03.401748 914 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } }
I20260810 14:58:03.401857 914 leader_election.cc:302] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40f0c5fc0b346bd984b23454280496a; no voters:
I20260810 14:58:03.402045 914 leader_election.cc:288] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:03.402082 920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:03.402231 920 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [term 1 LEADER]: Becoming Leader. State: Replica: e40f0c5fc0b346bd984b23454280496a, State: Running, Role: LEADER
I20260810 14:58:03.402318 914 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:03.402483 920 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } }
I20260810 14:58:03.403684 922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e40f0c5fc0b346bd984b23454280496a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } } }
I20260810 14:58:03.403789 922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:03.403898 924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40f0c5fc0b346bd984b23454280496a. Latest consensus state: current_term: 1 leader_uuid: "e40f0c5fc0b346bd984b23454280496a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f0c5fc0b346bd984b23454280496a" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40639 } } }
I20260810 14:58:03.404117 924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a [sys.catalog]: This master's current role is: LEADER
W20260810 14:58:03.404222 945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:03.404277 945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:03.404328 948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:03.404479 948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:03.406070 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.396031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.28.193.254:40639)
0810 14:58:03.396205 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:03.396214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:03.396239 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:03.396649 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:03.396656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:03.396701 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:03.396819 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:03.397432 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.398385 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.399168 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.399571 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.399663 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.402073 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:03.402079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:03.402079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:03.402092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:03.404508 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.404846 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.404849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.404849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.404943 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.405140 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.405142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.405142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.405457 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:03.405613 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:03.405743 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 14:58:03.405882 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260810 14:58:03.410094 948 catalog_manager.cc:1347] Generated new cluster ID: 749b61b690874f5883e952983ea1f4b6
I20260810 14:58:03.410148 948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:03.432818 948 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:03.433336 948 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:03.438069 948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40f0c5fc0b346bd984b23454280496a: Generated new TSK 0
I20260810 14:58:03.438246 948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:03.471930 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:03.552557 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:03.552752 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:03.552793 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:03.553953 955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:03.553994 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:03.554028 955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:03.555464 955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:03.555702 955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:03.555913 955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:03.560801 969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.560822 967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.560801 965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.560977 955 server_base.cc:1034] running on GCE node
I20260810 14:58:03.561343 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:03.561574 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:03.562740 955 hybrid_clock.cc:648] HybridClock initialized: now 1786373883562712 us; error 39 us; skew 500 ppm
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:03.565218 955 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:03.565656 955 webserver.cc:460] Webserver started at http://127.28.193.193:33113/ using document root <none> and password file <none>
I20260810 14:58:03.565909 955 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:03.565960 955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:03.566089 955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:03.567157 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "927583ea9aa24095ba4aa6ffe63b6103"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "1ddd9a4aa5eeaa7f41ab3bd36f13c5b2"
server_key_iv: "a4d74ee3b2da2e46114df5b2d7ae2483"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.567483 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "927583ea9aa24095ba4aa6ffe63b6103"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "1ddd9a4aa5eeaa7f41ab3bd36f13c5b2"
server_key_iv: "a4d74ee3b2da2e46114df5b2d7ae2483"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.568718 955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:03.569380 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.569540 955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:03.569628 955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "927583ea9aa24095ba4aa6ffe63b6103"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "1ddd9a4aa5eeaa7f41ab3bd36f13c5b2"
server_key_iv: "a4d74ee3b2da2e46114df5b2d7ae2483"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.569693 955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:03.604673 955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:03.606757 955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:03.606904 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:03.607138 955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:03.607483 955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:03.607519 955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.607548 955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:03.607564 955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.613067 955 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:41023
I20260810 14:58:03.613121 1158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:41023 every 8 connection(s)
I20260810 14:58:03.613442 955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:03.617532 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 955
I20260810 14:58:03.617609 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:03.617800 29447 external_mini_cluster.cc:1428] Setting key 37f7b0608fc480556b8111f94539ef98
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:03.622668 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.614203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33667 (local address 127.28.193.254:40639)
0810 14:58:03.614256 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:03.614258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:03.614467 (+ 209us) server_negotiation.cc:407] Connection header received
0810 14:58:03.614984 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:03.614985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:03.614998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:03.615047 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:03.616088 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.616586 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.617560 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.617834 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.617908 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.620063 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:03.620066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:03.620066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:03.620073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:03.621326 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.621687 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.621688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.621688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.621774 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.622066 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.622067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.622067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.622168 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:03.622225 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:03.622498 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 14:58:03.622579 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 14:58:03.622848 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.614160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40639 (local address 127.28.193.193:33667)
0810 14:58:03.614337 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:03.614383 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:03.614884 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:03.615131 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:03.615135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:03.615774 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:03.615981 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.615988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.616673 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.616675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.617472 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.617475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.617916 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.617917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.618045 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.618539 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:03.618548 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:03.619981 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:03.621402 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.621408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.621410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.621622 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.621832 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.621833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.621833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.621943 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.622228 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:03.622235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:03.622375 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 14:58:03.622463 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:58:03.622618 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260810 14:58:03.623296 1160 heartbeater.cc:347] Connected to a master server at 127.28.193.254:40639
I20260810 14:58:03.623389 1160 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:03.623567 1160 heartbeater.cc:511] Master 127.28.193.254:40639 requested a full tablet report, sending...
I20260810 14:58:03.624147 860 ts_manager.cc:194] Registered new tserver with Master: 927583ea9aa24095ba4aa6ffe63b6103 (127.28.193.193:41023)
I20260810 14:58:03.625058 860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:33667
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:03.642156 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:03.723242 1165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:03.723445 1165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:03.723467 1165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:03.724562 1165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:03.724596 1165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:03.724619 1165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:03.726084 1165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:03.726346 1165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:03.726600 1165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:03.731510 1176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.731544 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.731511 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.731712 1165 server_base.cc:1034] running on GCE node
I20260810 14:58:03.732008 1165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:03.732234 1165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:03.733383 1165 hybrid_clock.cc:648] HybridClock initialized: now 1786373883733370 us; error 27 us; skew 500 ppm
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:03.735772 1165 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:03.736208 1165 webserver.cc:460] Webserver started at http://127.28.193.194:39859/ using document root <none> and password file <none>
I20260810 14:58:03.736430 1165 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:03.736480 1165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:03.736608 1165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:03.737619 1165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "813c80d7308449f6b3922b06ef20977a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "f105b1c9304e2eeb6f3416db9a3f731c"
server_key_iv: "5e3cbd5fa53a1593b4302646872e10df"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.737993 1165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "813c80d7308449f6b3922b06ef20977a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "f105b1c9304e2eeb6f3416db9a3f731c"
server_key_iv: "5e3cbd5fa53a1593b4302646872e10df"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.739210 1165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:03.739848 1196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.739995 1165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.740077 1165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "813c80d7308449f6b3922b06ef20977a"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "f105b1c9304e2eeb6f3416db9a3f731c"
server_key_iv: "5e3cbd5fa53a1593b4302646872e10df"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.740140 1165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:03.751171 1165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:03.752724 1165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:03.752852 1165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:03.753055 1165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:03.753337 1165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:03.753371 1165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.753396 1165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:03.753413 1165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.759091 1165 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:39139
I20260810 14:58:03.759128 1377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:39139 every 8 connection(s)
I20260810 14:58:03.759426 1165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:03.767243 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1165
I20260810 14:58:03.767351 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:03.767549 29447 external_mini_cluster.cc:1428] Setting key db2f9be31a6404c1451e3cf1b0155936
I20260810 14:58:03.768036 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.760087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:57597 (local address 127.28.193.254:40639)
0810 14:58:03.760163 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:03.760165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:03.760336 (+ 171us) server_negotiation.cc:407] Connection header received
0810 14:58:03.760836 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:03.760837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:03.760847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:03.760886 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:03.761911 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.762404 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.763334 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.763517 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.763561 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.765610 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:03.765611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:03.765612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:03.765619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:03.766702 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.767083 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.767084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.767084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.767127 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.767423 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.767424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.767424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.767550 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:03.767617 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:03.767892 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 14:58:03.767973 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:58:03.768251 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.760055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40639 (local address 127.28.193.194:57597)
0810 14:58:03.760192 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:03.760229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:03.760741 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:03.760973 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:03.760977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:03.761548 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:03.761839 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.761844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.762479 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.762480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.763265 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.763267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.763571 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.763572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.763667 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.764132 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:03.764140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:03.765523 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:03.766766 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.766772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.766774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.767005 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.767177 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.767178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.767178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.767312 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.767617 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:03.767624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:03.767769 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 14:58:03.767861 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:58:03.768025 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260810 14:58:03.768723 1378 heartbeater.cc:347] Connected to a master server at 127.28.193.254:40639
I20260810 14:58:03.768821 1378 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:03.769006 1378 heartbeater.cc:511] Master 127.28.193.254:40639 requested a full tablet report, sending...
I20260810 14:58:03.769344 860 ts_manager.cc:194] Registered new tserver with Master: 813c80d7308449f6b3922b06ef20977a (127.28.193.194:39139)
I20260810 14:58:03.769706 860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:57597
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:03.790144 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:03.870770 1387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:03.870971 1387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:03.870992 1387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:03.872114 1387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:03.872152 1387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:03.872181 1387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:03.873611 1387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:03.873831 1387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:03.874037 1387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:03.878885 1395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.878969 1387 server_base.cc:1034] running on GCE node
W20260810 14:58:03.879046 1396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:03.878916 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:03.879277 1387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:03.879513 1387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:03.880695 1387 hybrid_clock.cc:648] HybridClock initialized: now 1786373883880674 us; error 34 us; skew 500 ppm
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:03.883200 1387 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:03.883627 1387 webserver.cc:460] Webserver started at http://127.28.193.195:37587/ using document root <none> and password file <none>
I20260810 14:58:03.883821 1387 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:03.883868 1387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:03.884009 1387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:03.885037 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "0f73880bd2034a6795847738ed77f82f"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "5945713f1cceb651df8003d741d29cd1"
server_key_iv: "6ae5a5d98a378dc61fc488cb8a20ebd8"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.885377 1387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "0f73880bd2034a6795847738ed77f82f"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "5945713f1cceb651df8003d741d29cd1"
server_key_iv: "6ae5a5d98a378dc61fc488cb8a20ebd8"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.886664 1387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:03.887334 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.887476 1387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:03.887557 1387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "0f73880bd2034a6795847738ed77f82f"
format_stamp: "Formatted at 2026-08-10 14:58:03 on dist-test-slave-cvwc"
server_key: "5945713f1cceb651df8003d741d29cd1"
server_key_iv: "6ae5a5d98a378dc61fc488cb8a20ebd8"
server_key_version: "encryptionkey@0"
I20260810 14:58:03.887616 1387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:03.903381 1387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:03.905922 1387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:03.906095 1387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:03.906338 1387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:03.906648 1387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:03.906682 1387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.906713 1387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:03.906729 1387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:03.912226 1387 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:40441
I20260810 14:58:03.912307 1582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:40441 every 8 connection(s)
I20260810 14:58:03.912588 1387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:03.914433 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1387
I20260810 14:58:03.914525 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:03.914683 29447 external_mini_cluster.cc:1428] Setting key 736f5b1536e49c7bf5aa29fd6bf8b6fb
Aug 10 14:58:03 dist-test-slave-cvwc krb5kdc[787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373883, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:03.921335 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.913295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:36465 (local address 127.28.193.254:40639)
0810 14:58:03.913350 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:03.913352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:03.913521 (+ 169us) server_negotiation.cc:407] Connection header received
0810 14:58:03.914073 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:03.914074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:03.914086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:03.914137 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:03.915209 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.915745 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.916791 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.916997 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.917052 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.919175 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:03.919176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:03.919177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:03.919183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:03.920255 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.920601 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.920602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.920602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.920649 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:03.920851 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:03.920851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:03.920851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:03.920953 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:03.921000 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:03.921211 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:58:03.921270 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:58:03.921501 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:03.913265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40639 (local address 127.28.193.195:36465)
0810 14:58:03.913417 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:03.913452 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:03.913989 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:03.914196 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:03.914200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:03.914866 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:03.915126 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.915133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.915822 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.915823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.916699 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:03.916702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:03.917062 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:03.917063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:03.917163 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:03.917626 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:03.917634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:03.919088 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:03.920319 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.920332 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.920334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.920545 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.920696 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:03.920697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:03.920697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:03.920806 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:03.920998 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:03.921001 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:03.921113 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:58:03.921189 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:58:03.921310 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260810 14:58:03.921934 1583 heartbeater.cc:347] Connected to a master server at 127.28.193.254:40639
I20260810 14:58:03.922019 1583 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:03.922185 1583 heartbeater.cc:511] Master 127.28.193.254:40639 requested a full tablet report, sending...
I20260810 14:58:03.922549 860 ts_manager.cc:194] Registered new tserver with Master: 0f73880bd2034a6795847738ed77f82f (127.28.193.195:40441)
I20260810 14:58:03.922907 860 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:36465
I20260810 14:58:03.926396 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:58:03.938244 918 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:03.937300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.28.193.254:40639)
0810 14:58:03.937360 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:03.937362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:03.937451 (+ 89us) server_negotiation.cc:407] Connection header received
0810 14:58:03.937536 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:03.937538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:03.937547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:03.937586 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:03.938169 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39724: BlockingRecv error: recv got EOF from 127.0.0.1:39724 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:03.939043 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 955
I20260810 14:58:03.957932 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1165
I20260810 14:58:03.970566 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1387
I20260810 14:58:03.984371 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 803
2026-08-10T14:58:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2000 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:04 dist-test-slave-cvwc krb5kdc[1600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:04 dist-test-slave-cvwc krb5kdc[1600](info): set up 2 sockets
Aug 10 14:58:04 dist-test-slave-cvwc krb5kdc[1600](info): commencing operation
krb5kdc: starting...
W20260810 14:58:05.144371 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:05.260627 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46131
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:05.341248 1619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:05.341449 1619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:05.341470 1619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:05.342610 1619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:05.342651 1619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:05.342664 1619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:05.342675 1619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:05.342685 1619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:05.344175 1619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:05.344403 1619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:05.344605 1619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:05.349378 1627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:05.349433 1628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:05.349426 1619 server_base.cc:1034] running on GCE node
W20260810 14:58:05.349395 1632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:05.349864 1619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:05.350139 1619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:05.351272 1619 hybrid_clock.cc:648] HybridClock initialized: now 1786373885351259 us; error 29 us; skew 500 ppm
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373885, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:05.353677 1619 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:05.354090 1619 webserver.cc:460] Webserver started at http://127.28.193.254:43883/ using document root <none> and password file <none>
I20260810 14:58:05.354283 1619 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:05.354322 1619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:05.354413 1619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:05.355412 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "7527f5fe1c97492b850de7486f74bafb"
format_stamp: "Formatted at 2026-08-10 14:58:05 on dist-test-slave-cvwc"
server_key: "2d800a8e4be9b8135e2bbe17e3d0e502"
server_key_iv: "05c8ef8ab10a231c1bd71eeaf8af03b7"
server_key_version: "encryptionkey@0"
I20260810 14:58:05.355744 1619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "7527f5fe1c97492b850de7486f74bafb"
format_stamp: "Formatted at 2026-08-10 14:58:05 on dist-test-slave-cvwc"
server_key: "2d800a8e4be9b8135e2bbe17e3d0e502"
server_key_iv: "05c8ef8ab10a231c1bd71eeaf8af03b7"
server_key_version: "encryptionkey@0"
I20260810 14:58:05.356915 1619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:05.357545 1646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:05.357692 1619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:05.357802 1619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "7527f5fe1c97492b850de7486f74bafb"
format_stamp: "Formatted at 2026-08-10 14:58:05 on dist-test-slave-cvwc"
server_key: "2d800a8e4be9b8135e2bbe17e3d0e502"
server_key_iv: "05c8ef8ab10a231c1bd71eeaf8af03b7"
server_key_version: "encryptionkey@0"
I20260810 14:58:05.357864 1619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:05.372643 1619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:05.374807 1619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:05.374956 1619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:05.378716 1619 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:46131
I20260810 14:58:05.378750 1733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:46131 every 8 connection(s)
I20260810 14:58:05.379287 1619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:05.379967 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:05.382265 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: Bootstrap starting.
I20260810 14:58:05.382808 1734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:05.383052 1734 log.cc:824] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:05.383665 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: No bootstrap required, opened a new log
I20260810 14:58:05.385134 1734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } }
I20260810 14:58:05.385275 1734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:05.385319 1734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7527f5fe1c97492b850de7486f74bafb, State: Initialized, Role: FOLLOWER
I20260810 14:58:05.385478 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1619
I20260810 14:58:05.385470 1734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } }
I20260810 14:58:05.385545 1734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:05.385565 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:05.385583 1734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:05.385632 1734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:05.385782 29447 external_mini_cluster.cc:1428] Setting key 07aa20a461c392397401943dc9facf28
I20260810 14:58:05.386682 1734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } }
I20260810 14:58:05.386797 1734 leader_election.cc:302] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7527f5fe1c97492b850de7486f74bafb; no voters:
I20260810 14:58:05.386967 1734 leader_election.cc:288] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:05.387009 1740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:05.387140 1740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [term 1 LEADER]: Becoming Leader. State: Replica: 7527f5fe1c97492b850de7486f74bafb, State: Running, Role: LEADER
I20260810 14:58:05.387231 1734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:05.387233 1740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } }
I20260810 14:58:05.388523 1742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7527f5fe1c97492b850de7486f74bafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } } }
I20260810 14:58:05.388669 1741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7527f5fe1c97492b850de7486f74bafb. Latest consensus state: current_term: 1 leader_uuid: "7527f5fe1c97492b850de7486f74bafb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7527f5fe1c97492b850de7486f74bafb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46131 } } }
I20260810 14:58:05.388801 1742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:05.388805 1741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb [sys.catalog]: This master's current role is: LEADER
W20260810 14:58:05.389174 1761 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:05.389221 1761 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:05.389262 1762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:05.389406 1762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:05.390940 1762 catalog_manager.cc:1347] Generated new cluster ID: 090de82adfce4431a7a0d0dce7ae01cb
I20260810 14:58:05.390993 1762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:05.396899 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:05.386229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.28.193.254:46131)
0810 14:58:05.386405 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:05.386415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:05.386441 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:58:05.386536 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:05.386542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:05.386584 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:05.386690 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:05.387477 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:05.388913 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:05.389981 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:05.390640 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:05.390745 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:05.393413 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:05.393417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:05.393417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:05.393429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:05.395359 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:05.395718 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:05.395721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:05.395721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:05.395843 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:05.396017 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:05.396018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:05.396018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:05.396322 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:05.396497 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:05.396565 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 14:58:05.396701 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260810 14:58:05.397897 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:58:05.400117 1762 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:05.400729 1762 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:05.408978 1762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7527f5fe1c97492b850de7486f74bafb: Generated new TSK 0
I20260810 14:58:05.409199 1762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:05.420718 1739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:05.417026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.28.193.254:46131)
0810 14:58:05.417081 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:05.417084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:05.417172 (+ 88us) server_negotiation.cc:407] Connection header received
0810 14:58:05.417218 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:05.417220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:05.417233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:05.417304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:05.417643 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:05.418502 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:05.419189 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:05.419515 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:05.419584 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:05.420236 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:05.420238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:05.420238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:05.420481 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:05.420501 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:05.420509 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:05.420569 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:05.420608 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 14:58:05.420635 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:05.422972 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1619
2026-08-10T14:58:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1441 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1778](info): set up 2 sockets
Aug 10 14:58:05 dist-test-slave-cvwc krb5kdc[1778](info): commencing operation
krb5kdc: starting...
W20260810 14:58:06.582136 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:06 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:06.698357 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42189
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:42189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:06.778996 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:06.779201 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:06.779232 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:06.780366 1808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:06.780408 1808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:06.780427 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:06.780444 1808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:06.780467 1808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:06.781963 1808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:42189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:06.782210 1808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:06.782416 1808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:06.787022 1808 server_base.cc:1034] running on GCE node
W20260810 14:58:06.787007 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:06.787017 1823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:06.786999 1818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:06.787449 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:06.787720 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:06.788857 1808 hybrid_clock.cc:648] HybridClock initialized: now 1786373886788846 us; error 34 us; skew 500 ppm
Aug 10 14:58:06 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:06.791221 1808 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:06.791594 1808 webserver.cc:460] Webserver started at http://127.28.193.254:43067/ using document root <none> and password file <none>
I20260810 14:58:06.791786 1808 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:06.791824 1808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:06.791911 1808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:06.792927 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "71adfd5c04f14172b8113c05bb14b2e6"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "f96c0e14ec732ac383eb639acdd7b3da"
server_key_iv: "0244442c230ad64291d8b4ac7a792588"
server_key_version: "encryptionkey@0"
I20260810 14:58:06.793237 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "71adfd5c04f14172b8113c05bb14b2e6"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "f96c0e14ec732ac383eb639acdd7b3da"
server_key_iv: "0244442c230ad64291d8b4ac7a792588"
server_key_version: "encryptionkey@0"
I20260810 14:58:06.794451 1808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:06.795085 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:06.795243 1808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:06.795324 1808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "71adfd5c04f14172b8113c05bb14b2e6"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "f96c0e14ec732ac383eb639acdd7b3da"
server_key_iv: "0244442c230ad64291d8b4ac7a792588"
server_key_version: "encryptionkey@0"
I20260810 14:58:06.795379 1808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:06.802711 1808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:06.804358 1808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:06.804486 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:06.808225 1808 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:42189
I20260810 14:58:06.808259 1920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:42189 every 8 connection(s)
I20260810 14:58:06.808583 1808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:06.809419 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:06.811661 1921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6: Bootstrap starting.
I20260810 14:58:06.812187 1921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:06.812445 1921 log.cc:824] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:06.812685 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1808
I20260810 14:58:06.812776 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:06.812928 29447 external_mini_cluster.cc:1428] Setting key d346243ec65900e9a9c149b0e7fd99f0
I20260810 14:58:06.813141 1921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6: No bootstrap required, opened a new log
I20260810 14:58:06.815738 1921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } }
I20260810 14:58:06.815873 1921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:06.815907 1921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71adfd5c04f14172b8113c05bb14b2e6, State: Initialized, Role: FOLLOWER
I20260810 14:58:06.816026 1921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } }
I20260810 14:58:06.816111 1921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:06.816143 1921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:06.816190 1921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:06.816848 1921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } }
I20260810 14:58:06.816958 1921 leader_election.cc:302] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71adfd5c04f14172b8113c05bb14b2e6; no voters:
I20260810 14:58:06.817108 1921 leader_election.cc:288] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:06.817145 1929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:06.817278 1929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [term 1 LEADER]: Becoming Leader. State: Replica: 71adfd5c04f14172b8113c05bb14b2e6, State: Running, Role: LEADER
I20260810 14:58:06.817365 1921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:06.817405 1929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } }
I20260810 14:58:06.817837 1936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71adfd5c04f14172b8113c05bb14b2e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } } }
I20260810 14:58:06.817962 1936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:06.817837 1929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71adfd5c04f14172b8113c05bb14b2e6. Latest consensus state: current_term: 1 leader_uuid: "71adfd5c04f14172b8113c05bb14b2e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71adfd5c04f14172b8113c05bb14b2e6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42189 } } }
I20260810 14:58:06.818084 1929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:06.818600 1943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:06.819188 1943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 14:58:06 dist-test-slave-cvwc krb5kdc[1778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:06.820704 1943 catalog_manager.cc:1347] Generated new cluster ID: b7ca7ebb9ee0466eab43c8652ac18752
I20260810 14:58:06.820760 1943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:06.823722 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:06.813388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.28.193.254:42189)
0810 14:58:06.813577 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:06.813586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:06.813688 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:58:06.814401 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:06.814407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:06.814456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:06.814565 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:06.815338 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:06.816293 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:06.817294 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:06.817963 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:06.818068 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:06.820742 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:06.820746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:06.820746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:06.820758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:06.822403 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:06.822708 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:06.822710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:06.822711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:06.822797 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:06.822968 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:06.822969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:06.822970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:06.823191 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:06.823357 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:06.823424 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:58:06.823546 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260810 14:58:06.842727 1943 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:06.843470 1943 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:06.849085 1943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71adfd5c04f14172b8113c05bb14b2e6: Generated new TSK 0
I20260810 14:58:06.849277 1943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:06.885655 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:06.987207 1961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:06.987396 1961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:06.987416 1961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:06.988520 1961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:06.988562 1961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:06.988587 1961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:06.990036 1961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:06.990270 1961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:06.990481 1961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:06.995393 1975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:06.995402 1971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:06.995510 1961 server_base.cc:1034] running on GCE node
W20260810 14:58:06.995476 1972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:06.995865 1961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:06.996098 1961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:06.997274 1961 hybrid_clock.cc:648] HybridClock initialized: now 1786373886997258 us; error 32 us; skew 500 ppm
Aug 10 14:58:06 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:06.999696 1961 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:07.000121 1961 webserver.cc:460] Webserver started at http://127.28.193.193:46175/ using document root <none> and password file <none>
I20260810 14:58:07.000310 1961 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:07.000356 1961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:07.000490 1961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:07.001477 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "1ad5e4f9cf214707a0b8d1e4fe8429d7"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "eb11c3962da2da1fe5b3d7fa9c10c934"
server_key_iv: "ca7f6684ed181c2b6ecedd06cb423791"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.001845 1961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "1ad5e4f9cf214707a0b8d1e4fe8429d7"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "eb11c3962da2da1fe5b3d7fa9c10c934"
server_key_iv: "ca7f6684ed181c2b6ecedd06cb423791"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.003096 1961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:07.003754 1985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.003906 1961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:07.003989 1961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "1ad5e4f9cf214707a0b8d1e4fe8429d7"
format_stamp: "Formatted at 2026-08-10 14:58:06 on dist-test-slave-cvwc"
server_key: "eb11c3962da2da1fe5b3d7fa9c10c934"
server_key_iv: "ca7f6684ed181c2b6ecedd06cb423791"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.004055 1961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:07.017144 1961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:07.019188 1961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:07.019320 1961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:07.019527 1961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:07.019825 1961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:07.019860 1961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.019887 1961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:07.019904 1961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.025346 1961 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:41695
I20260810 14:58:07.025406 2153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:41695 every 8 connection(s)
I20260810 14:58:07.025696 1961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373886, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:07.032189 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1961
I20260810 14:58:07.032297 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:07.032517 29447 external_mini_cluster.cc:1428] Setting key c13be9bc0788f035cf99fdd0b63ae31e
I20260810 14:58:07.035404 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.026354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:41359 (local address 127.28.193.254:42189)
0810 14:58:07.026427 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:07.026429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:07.026787 (+ 358us) server_negotiation.cc:407] Connection header received
0810 14:58:07.027464 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:07.027466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:07.027477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:07.027534 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:07.028544 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.029085 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.030159 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.030378 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.030434 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.032555 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:07.032557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:07.032557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:07.032564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:07.034037 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.034459 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.034460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.034460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.034506 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.034775 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.034776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.034777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.034917 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:07.034988 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:07.035216 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:58:07.035300 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:58:07.035523 2156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.026320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42189 (local address 127.28.193.193:41359)
0810 14:58:07.026477 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:07.026512 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:07.027277 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:07.027666 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:07.027670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:07.028254 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:07.028442 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.028447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.029176 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.029177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.030072 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.030074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.030454 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.030455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.030564 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.030990 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:07.030998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:07.032443 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:07.034108 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.034116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.034119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.034393 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.034561 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.034562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.034562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.034697 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.034984 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:07.034988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:07.035099 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:58:07.035182 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:58:07.035307 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260810 14:58:07.035967 2154 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42189
I20260810 14:58:07.036073 2154 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:07.036254 2154 heartbeater.cc:511] Master 127.28.193.254:42189 requested a full tablet report, sending...
I20260810 14:58:07.036767 1864 ts_manager.cc:194] Registered new tserver with Master: 1ad5e4f9cf214707a0b8d1e4fe8429d7 (127.28.193.193:41695)
I20260810 14:58:07.037352 1864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:41359
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:07.058579 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:07.139850 2159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:07.140084 2159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:07.140115 2159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:07.141292 2159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:07.141333 2159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:07.141368 2159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:07.142923 2159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:07.143172 2159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:07.143388 2159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:07.148306 2172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:07.148334 2166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:07.148334 2167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:07.148676 2159 server_base.cc:1034] running on GCE node
I20260810 14:58:07.148859 2159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:07.149060 2159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:07.150197 2159 hybrid_clock.cc:648] HybridClock initialized: now 1786373887150181 us; error 29 us; skew 500 ppm
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:07.156224 2159 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:07.156690 2159 webserver.cc:460] Webserver started at http://127.28.193.194:35657/ using document root <none> and password file <none>
I20260810 14:58:07.156921 2159 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:07.156965 2159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:07.157056 2159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:07.158147 2159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "d8e1471b295b40feac8953f4f9dd4abc"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "f762db022e6c32a8a71e3b565a305451"
server_key_iv: "7019dcdada1bc33861afb521cb4a680a"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.158464 2159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e1471b295b40feac8953f4f9dd4abc"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "f762db022e6c32a8a71e3b565a305451"
server_key_iv: "7019dcdada1bc33861afb521cb4a680a"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.159766 2159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:07.160485 2182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.160670 2159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:07.160748 2159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "d8e1471b295b40feac8953f4f9dd4abc"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "f762db022e6c32a8a71e3b565a305451"
server_key_iv: "7019dcdada1bc33861afb521cb4a680a"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.160820 2159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:07.172477 2159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:07.174353 2159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:07.174484 2159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:07.174716 2159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:07.175046 2159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:07.175081 2159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.175102 2159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:07.175127 2159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.180727 2159 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:45031
I20260810 14:58:07.180799 2360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:45031 every 8 connection(s)
I20260810 14:58:07.181100 2159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:07.183842 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2159
I20260810 14:58:07.183948 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:07.184122 29447 external_mini_cluster.cc:1428] Setting key dd48f128044618828d34117c701a7e7b
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:07.190451 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.181802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:59455 (local address 127.28.193.254:42189)
0810 14:58:07.181869 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:07.181872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:07.182072 (+ 200us) server_negotiation.cc:407] Connection header received
0810 14:58:07.182545 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:07.182546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:07.182556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:07.182606 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:07.183511 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.184106 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.185278 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.185474 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.185525 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.187704 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:07.187705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:07.187706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:07.187714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:07.189087 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.189492 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.189493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.189494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.189554 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.189806 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.189808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.189808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.189936 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:07.189994 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:07.190265 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 14:58:07.190360 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:58:07.190642 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.181789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42189 (local address 127.28.193.194:59455)
0810 14:58:07.181951 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:07.181989 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:07.182468 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:07.182664 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:07.182668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:07.183246 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:07.183442 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.183448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.184197 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.184198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.185199 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.185202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.185547 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.185548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.185682 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.186193 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:07.186202 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:07.187615 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:07.189185 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.189193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.189195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.189421 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.189608 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.189608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.189609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.189746 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.189992 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:07.189999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:07.190135 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 14:58:07.190220 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:58:07.190381 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260810 14:58:07.191113 2361 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42189
I20260810 14:58:07.191210 2361 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:07.191401 2361 heartbeater.cc:511] Master 127.28.193.254:42189 requested a full tablet report, sending...
I20260810 14:58:07.191793 1864 ts_manager.cc:194] Registered new tserver with Master: d8e1471b295b40feac8953f4f9dd4abc (127.28.193.194:45031)
I20260810 14:58:07.192339 1864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:59455
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:07.208081 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:07.288792 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:07.288995 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:07.289017 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:07.290150 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:07.290190 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:07.290215 2370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:07.291630 2370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:07.291854 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:07.292060 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:07.296885 2380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:07.296886 2385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:07.296926 2381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:07.297124 2370 server_base.cc:1034] running on GCE node
I20260810 14:58:07.297318 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:07.297535 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:07.298668 2370 hybrid_clock.cc:648] HybridClock initialized: now 1786373887298658 us; error 28 us; skew 500 ppm
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:07.300961 2370 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:07.301347 2370 webserver.cc:460] Webserver started at http://127.28.193.195:40079/ using document root <none> and password file <none>
I20260810 14:58:07.301569 2370 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:07.301618 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:07.301784 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:07.302776 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "394551e840834f1daf1f4a77723ed394"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "b4d09d0465b402f32a72e55dbc0af09f"
server_key_iv: "2ca4bd3e3e2a4d3b3eb2fe82710559a3"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.303093 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "394551e840834f1daf1f4a77723ed394"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "b4d09d0465b402f32a72e55dbc0af09f"
server_key_iv: "2ca4bd3e3e2a4d3b3eb2fe82710559a3"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.304271 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:07.304898 2396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.305040 2370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:07.305119 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "394551e840834f1daf1f4a77723ed394"
format_stamp: "Formatted at 2026-08-10 14:58:07 on dist-test-slave-cvwc"
server_key: "b4d09d0465b402f32a72e55dbc0af09f"
server_key_iv: "2ca4bd3e3e2a4d3b3eb2fe82710559a3"
server_key_version: "encryptionkey@0"
I20260810 14:58:07.305182 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:07.314884 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:07.316643 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:07.316766 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:07.316991 2370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:07.317277 2370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:07.317310 2370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.317337 2370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:07.317360 2370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:07.322925 2370 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:34403
I20260810 14:58:07.322965 2555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:34403 every 8 connection(s)
I20260810 14:58:07.323290 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:07.331349 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.323951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:43557 (local address 127.28.193.254:42189)
0810 14:58:07.323994 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:07.323996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:07.324188 (+ 192us) server_negotiation.cc:407] Connection header received
0810 14:58:07.324648 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:07.324649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:07.324659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:07.324704 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:07.325540 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.326097 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.327002 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.327175 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.327218 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.329092 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:07.329094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:07.329094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:07.329101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:07.330292 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.330644 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.330644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.330645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.330693 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.330885 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.330886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.330886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.330987 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:07.331030 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:07.331228 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 14:58:07.331285 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 14:58:07.331543 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.323925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42189 (local address 127.28.193.195:43557)
0810 14:58:07.324076 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:07.324112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:07.324572 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:07.324763 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:07.324766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:07.325289 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:07.325478 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.325483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.326165 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.326166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.326942 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:07.326944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.327226 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.327226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:07.327316 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.327714 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:07.327722 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:07.328996 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:07.330363 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.330369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.330371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.330581 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.330736 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:07.330736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:07.330737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:07.330831 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:07.331029 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:07.331032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:07.331137 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 14:58:07.331211 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:58:07.331334 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260810 14:58:07.331988 2556 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42189
I20260810 14:58:07.332078 2556 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:07.332247 2556 heartbeater.cc:511] Master 127.28.193.254:42189 requested a full tablet report, sending...
I20260810 14:58:07.332587 1864 ts_manager.cc:194] Registered new tserver with Master: 394551e840834f1daf1f4a77723ed394 (127.28.193.195:34403)
I20260810 14:58:07.332894 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2370
I20260810 14:58:07.332958 1864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:43557
I20260810 14:58:07.332983 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:07.333164 29447 external_mini_cluster.cc:1428] Setting key 9efab72e4f9e28d90058cf779620dab5
I20260810 14:58:07.333981 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[1778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:07.355381 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:07.349311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.28.193.254:42189)
0810 14:58:07.349355 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:07.349357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:07.349474 (+ 117us) server_negotiation.cc:407] Connection header received
0810 14:58:07.349555 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:07.349556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:07.349564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:07.349615 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:07.350202 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.350670 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.351338 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:07.351505 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:07.351545 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:07.353482 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:07.353484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:07.353484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:07.353490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:07.354563 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.354824 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.354825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.354825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.354881 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:07.355008 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:07.355009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:07.355009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:07.355128 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:07.355193 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:07.355260 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:58:07.355315 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 14:58:07.355590 1864 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41738
I20260810 14:58:07.355947 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1961
I20260810 14:58:07.373773 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2159
I20260810 14:58:07.382259 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2370
I20260810 14:58:07.395160 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1808
2026-08-10T14:58:07Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1969 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[2574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[2574](info): set up 2 sockets
Aug 10 14:58:07 dist-test-slave-cvwc krb5kdc[2574](info): commencing operation
krb5kdc: starting...
W20260810 14:58:08.552439 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:08 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:08.669215 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41173
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:41173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:08.749681 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:08.749917 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:08.749948 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:08.751085 2592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:08.751125 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:08.751143 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:08.751159 2592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:08.751173 2592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:08.752645 2592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:41173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:08.752882 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:08.753095 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:08.757763 2605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:08.757797 2603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:08.757828 2608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:08.757977 2592 server_base.cc:1034] running on GCE node
I20260810 14:58:08.758211 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:08.758451 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:08.759620 2592 hybrid_clock.cc:648] HybridClock initialized: now 1786373888759597 us; error 36 us; skew 500 ppm
Aug 10 14:58:08 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:08.762126 2592 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:08.762537 2592 webserver.cc:460] Webserver started at http://127.28.193.254:40097/ using document root <none> and password file <none>
I20260810 14:58:08.762730 2592 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:08.762809 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:08.762930 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:08.764010 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "e9da599ca6934f45982c523751f3acff"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "16c4bb9b40a99ce139e27439f5ae8e00"
server_key_iv: "a498e6d427c22c759c9ab80b3aebf58f"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.764351 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "e9da599ca6934f45982c523751f3acff"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "16c4bb9b40a99ce139e27439f5ae8e00"
server_key_iv: "a498e6d427c22c759c9ab80b3aebf58f"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.765621 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:08.766368 2617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:08.766531 2592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:08.766616 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "e9da599ca6934f45982c523751f3acff"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "16c4bb9b40a99ce139e27439f5ae8e00"
server_key_iv: "a498e6d427c22c759c9ab80b3aebf58f"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.766676 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:08.782415 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:08.784415 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:08.784548 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:08.788239 2592 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:41173
I20260810 14:58:08.788285 2695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:41173 every 8 connection(s)
I20260810 14:58:08.788724 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:08.789366 2697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:08.791567 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff: Bootstrap starting.
I20260810 14:58:08.792084 2697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:08.792316 2697 log.cc:824] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:08.792932 2697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff: No bootstrap required, opened a new log
I20260810 14:58:08.794036 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2592
I20260810 14:58:08.794140 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:08.794304 29447 external_mini_cluster.cc:1428] Setting key 3cee91b16a83b6cb13c85e13df84a42a
I20260810 14:58:08.794534 2697 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } }
I20260810 14:58:08.794716 2697 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:08.794765 2697 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9da599ca6934f45982c523751f3acff, State: Initialized, Role: FOLLOWER
I20260810 14:58:08.794909 2697 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } }
I20260810 14:58:08.794996 2697 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:08.795037 2697 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:08.795091 2697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:08.795713 2697 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } }
I20260810 14:58:08.795830 2697 leader_election.cc:302] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9da599ca6934f45982c523751f3acff; no voters:
I20260810 14:58:08.795981 2697 leader_election.cc:288] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:08.796007 2702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:08.796113 2702 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [term 1 LEADER]: Becoming Leader. State: Replica: e9da599ca6934f45982c523751f3acff, State: Running, Role: LEADER
I20260810 14:58:08.796195 2702 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } }
I20260810 14:58:08.796231 2697 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:08.796512 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9da599ca6934f45982c523751f3acff. Latest consensus state: current_term: 1 leader_uuid: "e9da599ca6934f45982c523751f3acff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } } }
I20260810 14:58:08.796587 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:08.796775 2712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:08.797303 2712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:08.798105 2706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9da599ca6934f45982c523751f3acff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9da599ca6934f45982c523751f3acff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41173 } } }
I20260810 14:58:08.798350 2706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:08.799500 2712 catalog_manager.cc:1347] Generated new cluster ID: aa372e694c124abfb7fed81440f81149
I20260810 14:58:08.799552 2712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:08 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:08.806655 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:08.794790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.28.193.254:41173)
0810 14:58:08.795414 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:08.795424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:08.795459 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:58:08.795575 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:08.795582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:08.795638 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:08.795799 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:08.796520 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.797755 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.798731 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.799281 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.799383 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:08.801978 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:08.801981 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:08.801982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:08.801994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:08.804657 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:08.805075 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:08.805078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:08.805079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:08.805200 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:08.805453 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:08.805454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:08.805454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:08.805890 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:08.806049 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:08.806188 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:58:08.806369 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":525,"threads_started":1}
I20260810 14:58:08.812630 2712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:08.813074 2712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:08.823060 2712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9da599ca6934f45982c523751f3acff: Generated new TSK 0
I20260810 14:58:08.823203 2712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:08.860996 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:08.942987 2738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:08.943221 2738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:08.943243 2738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:08.944371 2738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:08.944410 2738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:08.944434 2738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:08.945957 2738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:08.946179 2738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:08.946409 2738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:08.951227 2747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:08.951233 2746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:08.951233 2750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:08.951444 2738 server_base.cc:1034] running on GCE node
I20260810 14:58:08.951632 2738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:08.951830 2738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:08.952996 2738 hybrid_clock.cc:648] HybridClock initialized: now 1786373888952979 us; error 27 us; skew 500 ppm
Aug 10 14:58:08 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:08.955760 2738 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:08.956208 2738 webserver.cc:460] Webserver started at http://127.28.193.193:43709/ using document root <none> and password file <none>
I20260810 14:58:08.956405 2738 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:08.956450 2738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:08.956547 2738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:08.957648 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "3821b02252de486aa90b13957bdc8160"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "569d93e1059a50a1ebb7b11e031c569d"
server_key_iv: "a7bd9efc51b0db083763efcc50d21675"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.958034 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "3821b02252de486aa90b13957bdc8160"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "569d93e1059a50a1ebb7b11e031c569d"
server_key_iv: "a7bd9efc51b0db083763efcc50d21675"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.959367 2738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:08.960028 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:08.960165 2738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:08.960259 2738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "3821b02252de486aa90b13957bdc8160"
format_stamp: "Formatted at 2026-08-10 14:58:08 on dist-test-slave-cvwc"
server_key: "569d93e1059a50a1ebb7b11e031c569d"
server_key_iv: "a7bd9efc51b0db083763efcc50d21675"
server_key_version: "encryptionkey@0"
I20260810 14:58:08.960333 2738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:08.983170 2738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:08.985494 2738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:08.985654 2738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:08.985934 2738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:08.986243 2738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:08.986279 2738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:08.986303 2738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:08.986322 2738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:08.992426 2738 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45429
I20260810 14:58:08.992483 2937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45429 every 8 connection(s)
I20260810 14:58:08.992800 2738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:08.997306 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2738
I20260810 14:58:08.997411 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:08.997619 29447 external_mini_cluster.cc:1428] Setting key 7cb7b9cb2fb07a8bc19d9b3429367cb7
Aug 10 14:58:08 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373888, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:09.002758 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:08.993497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:45163 (local address 127.28.193.254:41173)
0810 14:58:08.993579 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:08.993581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:08.993761 (+ 180us) server_negotiation.cc:407] Connection header received
0810 14:58:08.994762 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:08.994763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:08.994778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:08.994828 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:08.995442 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.995982 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.997097 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.997332 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.997390 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:08.999913 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:08.999915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:08.999915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:08.999925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:09.001384 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.001783 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.001785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.001785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.001856 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.002100 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.002101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.002101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.002248 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:09.002336 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:09.002566 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:58:09.002659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:58:09.002818 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:08.993447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41173 (local address 127.28.193.193:45163)
0810 14:58:08.993623 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:08.993661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:08.994654 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:08.994888 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:08.994891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:08.995140 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:08.995364 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:08.995370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.996068 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.996069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:08.997011 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:08.997015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:08.997517 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:08.997518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:08.997638 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:08.998181 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:08.998189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:08.999797 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:09.001465 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.001471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.001473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.001683 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.001920 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.001920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.001921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.002032 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.002306 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:09.002308 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:09.002417 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 14:58:09.002492 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:58:09.002611 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260810 14:58:09.003366 2938 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41173
I20260810 14:58:09.003468 2938 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:09.003643 2938 heartbeater.cc:511] Master 127.28.193.254:41173 requested a full tablet report, sending...
I20260810 14:58:09.004139 2640 ts_manager.cc:194] Registered new tserver with Master: 3821b02252de486aa90b13957bdc8160 (127.28.193.193:45429)
I20260810 14:58:09.004698 2640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:45163
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:09.024034 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:09.122426 2944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:09.122620 2944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:09.122641 2944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:09.123713 2944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:09.123744 2944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:09.123768 2944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:09.125196 2944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:09.125380 2944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:09.125560 2944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:09.130419 2950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:09.130438 2949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:09.130409 2952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:09.130664 2944 server_base.cc:1034] running on GCE node
I20260810 14:58:09.130877 2944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:09.131093 2944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:09.132237 2944 hybrid_clock.cc:648] HybridClock initialized: now 1786373889132218 us; error 31 us; skew 500 ppm
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:09.134737 2944 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:09.135128 2944 webserver.cc:460] Webserver started at http://127.28.193.194:43551/ using document root <none> and password file <none>
I20260810 14:58:09.135337 2944 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:09.135385 2944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:09.135515 2944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:09.136515 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "0da60d9bd9184f1fa7acd414f8b2debd"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "16cff86b67456290aa736d355e785aa4"
server_key_iv: "bee03d9f884a1b6f1869d57a3cdb55b1"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.136842 2944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "0da60d9bd9184f1fa7acd414f8b2debd"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "16cff86b67456290aa736d355e785aa4"
server_key_iv: "bee03d9f884a1b6f1869d57a3cdb55b1"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.138137 2944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:09.138851 2961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.139008 2944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:09.139088 2944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "0da60d9bd9184f1fa7acd414f8b2debd"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "16cff86b67456290aa736d355e785aa4"
server_key_iv: "bee03d9f884a1b6f1869d57a3cdb55b1"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.139148 2944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:09.170560 2944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:09.172564 2944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:09.172703 2944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:09.172945 2944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:09.173305 2944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:09.173341 2944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.173372 2944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:09.173389 2944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.178958 2944 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44415
I20260810 14:58:09.179000 3131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44415 every 8 connection(s)
I20260810 14:58:09.179312 2944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:09.180154 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2944
I20260810 14:58:09.180262 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:09.180471 29447 external_mini_cluster.cc:1428] Setting key 3ce5d2414d6f48ba8059471f7452708e
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:09.188584 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.179993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:35263 (local address 127.28.193.254:41173)
0810 14:58:09.180076 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:09.180078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:09.180287 (+ 209us) server_negotiation.cc:407] Connection header received
0810 14:58:09.181475 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:09.181475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:09.181485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:09.181523 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:09.182345 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.182884 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.183852 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.184081 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.184137 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.186104 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:09.186106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:09.186106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:09.186113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:09.187341 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.187698 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.187699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.187700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.187752 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.187975 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.187976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.187976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.188085 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:09.188143 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:09.188404 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 14:58:09.188495 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:58:09.188740 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.179968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41173 (local address 127.28.193.194:35263)
0810 14:58:09.180148 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:09.180198 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:09.181371 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:09.181596 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:09.181601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:09.181994 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:09.182257 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:09.182264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.182981 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.182983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:09.183766 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:09.183768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.184143 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.184143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:09.184239 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.184675 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:09.184682 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:09.186016 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:09.187415 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.187421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.187423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.187631 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.187806 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.187807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.187808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.187918 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.188141 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:09.188148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:09.188284 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 14:58:09.188375 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:58:09.188527 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 14:58:09.189394 3132 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41173
I20260810 14:58:09.189508 3132 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:09.189673 3132 heartbeater.cc:511] Master 127.28.193.254:41173 requested a full tablet report, sending...
I20260810 14:58:09.190073 2640 ts_manager.cc:194] Registered new tserver with Master: 0da60d9bd9184f1fa7acd414f8b2debd (127.28.193.194:44415)
I20260810 14:58:09.190431 2640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:35263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:09.204037 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:09.284694 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:09.284888 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:09.284909 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:09.286109 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:09.286149 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:09.286172 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:09.287613 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:09.287791 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:09.287971 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:09.292542 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:09.292667 3138 server_base.cc:1034] running on GCE node
W20260810 14:58:09.292558 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:09.292613 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:09.293010 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:09.293222 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:09.294353 3138 hybrid_clock.cc:648] HybridClock initialized: now 1786373889294341 us; error 28 us; skew 500 ppm
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:09.296778 3138 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:09.297223 3138 webserver.cc:460] Webserver started at http://127.28.193.195:38947/ using document root <none> and password file <none>
I20260810 14:58:09.297411 3138 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:09.297451 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:09.297540 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:09.298632 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "588209cbb05c4137b23e560691b4e1ce"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "abad0f74be036708b5585f9a80620fa5"
server_key_iv: "91fbb92279ade0a6dd4a005706b44d8c"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.299010 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "588209cbb05c4137b23e560691b4e1ce"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "abad0f74be036708b5585f9a80620fa5"
server_key_iv: "91fbb92279ade0a6dd4a005706b44d8c"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.300180 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:09.300800 3167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.300944 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:09.301021 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "588209cbb05c4137b23e560691b4e1ce"
format_stamp: "Formatted at 2026-08-10 14:58:09 on dist-test-slave-cvwc"
server_key: "abad0f74be036708b5585f9a80620fa5"
server_key_iv: "91fbb92279ade0a6dd4a005706b44d8c"
server_key_version: "encryptionkey@0"
I20260810 14:58:09.301092 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:09.324740 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:09.326506 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:09.326634 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:09.326869 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:09.327191 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:09.327225 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.327247 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:09.327270 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:09.332751 3138 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:34807
I20260810 14:58:09.332792 3328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:34807 every 8 connection(s)
I20260810 14:58:09.333086 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:09.338932 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 3138
I20260810 14:58:09.339016 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:09.339182 29447 external_mini_cluster.cc:1428] Setting key 8187255e94294d229f7275b0aa48258f
I20260810 14:58:09.341867 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.333788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:39793 (local address 127.28.193.254:41173)
0810 14:58:09.333844 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:09.333846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:09.334046 (+ 200us) server_negotiation.cc:407] Connection header received
0810 14:58:09.334958 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:09.334959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:09.334969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:09.335024 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:09.335602 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.336098 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.337004 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.337180 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.337224 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.339337 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:09.339338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:09.339339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:09.339347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:09.340544 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.340919 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.340921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.340921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.340992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.341208 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.341209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.341210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.341321 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:09.341377 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:09.341673 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 14:58:09.341772 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:09.342018 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.333754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41173 (local address 127.28.193.195:39793)
0810 14:58:09.333919 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:09.333961 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:09.334873 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:09.335081 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:09.335085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:09.335337 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:09.335536 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:09.335540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.336164 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.336164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:09.336938 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:09.336940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.337236 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.337236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:09.337331 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.337802 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:09.337811 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:09.339240 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:09.340616 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.340624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.340626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.340851 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.341047 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:09.341048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:09.341048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:09.341151 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:09.341378 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:09.341384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:09.341516 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 14:58:09.341629 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 14:58:09.341807 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260810 14:58:09.342412 3329 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41173
I20260810 14:58:09.342507 3329 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:09.342710 3329 heartbeater.cc:511] Master 127.28.193.254:41173 requested a full tablet report, sending...
I20260810 14:58:09.343035 2640 ts_manager.cc:194] Registered new tserver with Master: 588209cbb05c4137b23e560691b4e1ce (127.28.193.195:34807)
I20260810 14:58:09.343396 2640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:39793
I20260810 14:58:09.351130 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:58:09.374527 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.366789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.28.193.193:45429)
0810 14:58:09.366974 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:09.366979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:09.366990 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:09.367018 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:09.367020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:09.367036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:09.367113 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:09.367672 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.368452 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.369224 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.369588 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.369653 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.371946 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:09.371953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:09.371953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:09.371959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:09.373405 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.373774 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.373778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.373778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.373852 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.374010 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.374011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.374011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.374144 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:09.374257 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:09.374365 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 14:58:09.374434 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[2574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:09.382159 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:09.375807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.28.193.254:41173)
0810 14:58:09.375853 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:09.375855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:09.375865 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:09.375919 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:09.375920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:09.375929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:09.375968 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:09.376350 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.376839 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.377446 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:09.377617 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:09.377657 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:09.379663 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:09.379665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:09.379665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:09.379672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:09.380988 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.381318 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.381319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.381319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.381374 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:09.381552 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:09.381553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:09.381553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:09.381702 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:09.381869 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:09.381988 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:58:09.382068 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 14:58:09.382402 2640 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44156
I20260810 14:58:09.382735 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2738
I20260810 14:58:09.397270 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2944
I20260810 14:58:09.410197 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 3138
I20260810 14:58:09.418476 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2592
2026-08-10T14:58:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2021 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[3348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[3348](info): set up 2 sockets
Aug 10 14:58:09 dist-test-slave-cvwc krb5kdc[3348](info): commencing operation
krb5kdc: starting...
W20260810 14:58:10.581758 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:10 dist-test-slave-cvwc krb5kdc[3348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:10.696249 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46563
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:46563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:10.775873 3367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:10.776065 3367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:10.776096 3367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:10.777204 3367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:10.777243 3367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:10.777261 3367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:10.777277 3367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:10.777292 3367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:10.778831 3367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:46563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:10.779074 3367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:10.779269 3367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:10.783898 3373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:10.783933 3374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:10.783933 3376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:10.784037 3367 server_base.cc:1034] running on GCE node
I20260810 14:58:10.784317 3367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:10.784601 3367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:10.785768 3367 hybrid_clock.cc:648] HybridClock initialized: now 1786373890785752 us; error 32 us; skew 500 ppm
Aug 10 14:58:10 dist-test-slave-cvwc krb5kdc[3348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373890, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:10.788025 3367 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:10.788326 3367 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:10.788372 3367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:10.788470 3367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:10.789407 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "c43f88e10a594fb8a1565b36aa4f3733"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "df4aa81fbb81663310970853727570c9"
server_key_iv: "3f4cfa89329af72d46461ff7e7fb1508"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.789692 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "c43f88e10a594fb8a1565b36aa4f3733"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "df4aa81fbb81663310970853727570c9"
server_key_iv: "3f4cfa89329af72d46461ff7e7fb1508"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.790865 3367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:10.791541 3382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:10.791683 3367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:10.791760 3367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "c43f88e10a594fb8a1565b36aa4f3733"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "df4aa81fbb81663310970853727570c9"
server_key_iv: "3f4cfa89329af72d46461ff7e7fb1508"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.791826 3367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:10.804883 3367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:10.806723 3367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:10.806836 3367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:10.810465 3367 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:46563
I20260810 14:58:10.810535 3465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:46563 every 8 connection(s)
I20260810 14:58:10.810739 3367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:10.811744 3466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:10.813975 3466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: Bootstrap starting.
I20260810 14:58:10.814527 3466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:10.814779 3466 log.cc:824] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:10.815461 3466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: No bootstrap required, opened a new log
I20260810 14:58:10.817111 3466 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } }
I20260810 14:58:10.817250 3466 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:10.817291 3466 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43f88e10a594fb8a1565b36aa4f3733, State: Initialized, Role: FOLLOWER
I20260810 14:58:10.817425 3466 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } }
I20260810 14:58:10.817489 3466 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:10.817519 3466 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:10.817565 3466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:10.818396 3466 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } }
I20260810 14:58:10.818503 3466 leader_election.cc:302] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43f88e10a594fb8a1565b36aa4f3733; no voters:
I20260810 14:58:10.818658 3466 leader_election.cc:288] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:10.818702 3475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:10.818852 3475 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [term 1 LEADER]: Becoming Leader. State: Replica: c43f88e10a594fb8a1565b36aa4f3733, State: Running, Role: LEADER
I20260810 14:58:10.818893 3466 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:10.818950 3475 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } }
I20260810 14:58:10.819283 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43f88e10a594fb8a1565b36aa4f3733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } } }
I20260810 14:58:10.819291 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43f88e10a594fb8a1565b36aa4f3733. Latest consensus state: current_term: 1 leader_uuid: "c43f88e10a594fb8a1565b36aa4f3733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f88e10a594fb8a1565b36aa4f3733" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46563 } } }
I20260810 14:58:10.819388 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:10.819391 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:10.819581 3488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:10.820083 3488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:10.820698 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 3367
I20260810 14:58:10.820804 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:10.820979 29447 external_mini_cluster.cc:1428] Setting key f560823591ab4c193abd2279585f5ae3
I20260810 14:58:10.821704 3488 catalog_manager.cc:1347] Generated new cluster ID: 1e17980a67be4ad190c863be1df3b60c
I20260810 14:58:10.821815 3488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:10 dist-test-slave-cvwc krb5kdc[3348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:10.832361 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:10.821562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.28.193.254:46563)
0810 14:58:10.821801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:10.821811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:10.821844 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:58:10.821958 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:10.821964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:10.822018 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:10.822146 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:10.822888 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:10.824243 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:10.825297 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:10.825925 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:10.826025 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:10.828722 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:10.828726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:10.828727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:10.828739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:10.830804 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:10.831158 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:10.831160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:10.831161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:10.831269 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:10.831442 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:10.831442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:10.831442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:10.831760 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:10.831923 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:10.831988 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 14:58:10.832124 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":73,"threads_started":1}
I20260810 14:58:10.835673 3488 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:10.836140 3488 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:10.839955 3488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: Generated new TSK 0
I20260810 14:58:10.840095 3488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:10.874567 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:10.978587 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:10.978781 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:10.978803 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:10.979944 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:10.979983 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:10.980008 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:10.981447 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:10.981622 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:10.981802 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:10.986408 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:10.986459 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:10.986601 3508 server_base.cc:1034] running on GCE node
W20260810 14:58:10.986608 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:10.986856 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:10.987082 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:10.988225 3508 hybrid_clock.cc:648] HybridClock initialized: now 1786373890988210 us; error 28 us; skew 500 ppm
Aug 10 14:58:10 dist-test-slave-cvwc krb5kdc[3348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373890, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:10.990618 3508 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:10.990893 3508 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:10.990934 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:10.991020 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:10.991966 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "1947deb942a946cb9c4d44faf1c0e4cf"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "5e4d5ab5ef08c036f0be88df1d1e02c8"
server_key_iv: "0aeeb3f32130ff69aa7f719de0553c48"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.992254 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "1947deb942a946cb9c4d44faf1c0e4cf"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "5e4d5ab5ef08c036f0be88df1d1e02c8"
server_key_iv: "0aeeb3f32130ff69aa7f719de0553c48"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.993436 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:10.994149 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:10.994313 3508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:10.994405 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "1947deb942a946cb9c4d44faf1c0e4cf"
format_stamp: "Formatted at 2026-08-10 14:58:10 on dist-test-slave-cvwc"
server_key: "5e4d5ab5ef08c036f0be88df1d1e02c8"
server_key_iv: "0aeeb3f32130ff69aa7f719de0553c48"
server_key_version: "encryptionkey@0"
I20260810 14:58:10.994472 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.011276 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.012861 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.012992 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.013190 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:11.013468 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:11.013500 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.013526 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:11.013552 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.018925 3508 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:39409
I20260810 14:58:11.018929 3701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:39409 every 8 connection(s)
I20260810 14:58:11.019264 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:11.020434 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 3508
I20260810 14:58:11.020524 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:11.020727 29447 external_mini_cluster.cc:1428] Setting key 7467709fc522ea1cda94a2f5373428e2
Aug 10 14:58:11 dist-test-slave-cvwc krb5kdc[3348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373890, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:11.028673 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.019884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:58985 (local address 127.28.193.254:46563)
0810 14:58:11.019925 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.019927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.020131 (+ 204us) server_negotiation.cc:407] Connection header received
0810 14:58:11.020645 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.020647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.020665 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.020734 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.021598 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.022266 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.023397 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.023600 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.023652 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.026021 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.026022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.026023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:11.026031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.027356 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.027794 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.027794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.027795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.027853 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.028098 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.028099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.028099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.028192 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.028236 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.028533 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.028608 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 14:58:11.028910 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.019870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46563 (local address 127.28.193.193:58985)
0810 14:58:11.020011 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.020056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.020548 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.020797 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.020801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.021067 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:11.021508 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.021513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.022358 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.022359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.023313 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.023317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.023673 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.023674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.023805 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.024249 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:11.024258 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:11.025914 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:11.027441 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.027447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.027449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.027720 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.027908 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.027909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.027909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.028039 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.028245 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:11.028251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:11.028407 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.028523 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.028679 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 14:58:11.029471 3702 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46563
I20260810 14:58:11.029567 3702 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:11.029736 3702 heartbeater.cc:511] Master 127.28.193.254:46563 requested a full tablet report, sending...
I20260810 14:58:11.030254 3408 ts_manager.cc:194] Registered new tserver with Master: 1947deb942a946cb9c4d44faf1c0e4cf (127.28.193.193:39409)
I20260810 14:58:11.030933 3408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:58985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:11.046166 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:11.125954 3707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:11.126147 3707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:11.126168 3707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:11.127297 3707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:11.127338 3707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:11.127368 3707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:11.128880 3707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:11.129101 3707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:11.129307 3707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:11.134199 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.134222 3717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.134192 3716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.134301 3707 server_base.cc:1034] running on GCE node
I20260810 14:58:11.134655 3707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:11.134891 3707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:11.136024 3707 hybrid_clock.cc:648] HybridClock initialized: now 1786373891136007 us; error 33 us; skew 500 ppm
Aug 10 14:58:11 dist-test-slave-cvwc krb5kdc[3348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373891, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:11.138393 3707 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:11.138757 3707 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:11.138811 3707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:11.138939 3707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:11.139937 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "9ad2915a63d44babb65e1fd1a1e1d50d"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b6057c417110d9306a65af545c69986"
server_key_iv: "cf1327a2237e6e990fb5f35a13cd1f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.140275 3707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "9ad2915a63d44babb65e1fd1a1e1d50d"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b6057c417110d9306a65af545c69986"
server_key_iv: "cf1327a2237e6e990fb5f35a13cd1f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.141479 3707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:11.142165 3728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.142318 3707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.142402 3707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "9ad2915a63d44babb65e1fd1a1e1d50d"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b6057c417110d9306a65af545c69986"
server_key_iv: "cf1327a2237e6e990fb5f35a13cd1f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.142481 3707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.161536 3707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.163710 3707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.163851 3707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.164076 3707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:11.164381 3707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:11.164415 3707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.164445 3707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:11.164479 3707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.169919 3707 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:36853
I20260810 14:58:11.169961 3894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:36853 every 8 connection(s)
I20260810 14:58:11.170218 3707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:11.170883 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 3707
I20260810 14:58:11.170997 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:11.171209 29447 external_mini_cluster.cc:1428] Setting key b14a7dee3d3b27b92c8c70df6fecb3ac
Aug 10 14:58:11 dist-test-slave-cvwc krb5kdc[3348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373891, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 14:58:11.179920 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.170915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:39401 (local address 127.28.193.254:46563)
0810 14:58:11.170979 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.170981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.171206 (+ 225us) server_negotiation.cc:407] Connection header received
0810 14:58:11.171816 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.171816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.171835 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.171886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.173047 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.173570 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.174616 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.174834 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.174895 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.177113 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.177114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.177115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:11.177122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.178520 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.178952 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.178953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.178953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.179000 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.179249 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.179249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.179249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.179346 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.179390 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.179728 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.179809 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:58:11.180088 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.170893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46563 (local address 127.28.193.194:39401)
0810 14:58:11.171063 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.171114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.171665 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.171974 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.171979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.172316 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:11.172959 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.172967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.173669 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.173671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.174523 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.174526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.174913 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.174914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.175046 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.175514 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:11.175522 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:11.177002 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:11.178599 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.178606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.178609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.178881 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.179054 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.179055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.179055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.179188 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.179400 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:11.179406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:11.179569 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.179683 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.179845 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 14:58:11.180594 3896 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46563
I20260810 14:58:11.180701 3896 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:11.180848 3896 heartbeater.cc:511] Master 127.28.193.254:46563 requested a full tablet report, sending...
I20260810 14:58:11.181237 3408 ts_manager.cc:194] Registered new tserver with Master: 9ad2915a63d44babb65e1fd1a1e1d50d (127.28.193.194:36853)
I20260810 14:58:11.181603 3408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:39401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:11.195655 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:11.274752 3903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:11.274950 3903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:11.274981 3903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:11.276150 3903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:11.276194 3903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:11.276229 3903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:11.277742 3903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:11.277982 3903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:11.278201 3903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:11.283066 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.283062 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.283061 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.283259 3903 server_base.cc:1034] running on GCE node
I20260810 14:58:11.283499 3903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:11.283717 3903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:11.284852 3903 hybrid_clock.cc:648] HybridClock initialized: now 1786373891284838 us; error 30 us; skew 500 ppm
Aug 10 14:58:11 dist-test-slave-cvwc krb5kdc[3348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373891, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:11.287168 3903 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:11.287509 3903 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:11.287559 3903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:11.287654 3903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:11.288645 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "bfe739550eda434cad631625ee1a697b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0ab328daa53987c9973e6a9db9e2c76a"
server_key_iv: "6b3eee6b60fc8df2324bab60b3ba8e4b"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.288939 3903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe739550eda434cad631625ee1a697b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0ab328daa53987c9973e6a9db9e2c76a"
server_key_iv: "6b3eee6b60fc8df2324bab60b3ba8e4b"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.290120 3903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:11.290746 3921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.290900 3903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:11.290982 3903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "bfe739550eda434cad631625ee1a697b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0ab328daa53987c9973e6a9db9e2c76a"
server_key_iv: "6b3eee6b60fc8df2324bab60b3ba8e4b"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.291035 3903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.305096 3903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.307121 3903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.307230 3903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.307416 3903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:11.307679 3903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:11.307709 3903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.307729 3903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:11.307740 3903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.312985 3903 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:40125
I20260810 14:58:11.313050 4080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:40125 every 8 connection(s)
I20260810 14:58:11.313251 3903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:11 dist-test-slave-cvwc krb5kdc[3348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373891, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:11.321264 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 3903
I20260810 14:58:11.321369 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:11.321561 29447 external_mini_cluster.cc:1428] Setting key 209902f08f13ade3bd1440b793c8ed40
I20260810 14:58:11.322060 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.313903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:58369 (local address 127.28.193.254:46563)
0810 14:58:11.313964 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.313966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.314157 (+ 191us) server_negotiation.cc:407] Connection header received
0810 14:58:11.314659 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.314660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.314673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.314720 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.315618 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.316118 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.317077 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.317279 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.317333 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.319456 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.319458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.319458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:11.319466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.320686 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.321045 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.321046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.321047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.321115 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.321386 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.321387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.321387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.321521 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.321581 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.321892 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.321972 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:11.322212 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.313883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46563 (local address 127.28.193.195:58369)
0810 14:58:11.314048 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.314083 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.314566 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.314804 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.314808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.315049 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:11.315527 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.315532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.316201 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.316202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.316996 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.316999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.317346 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.317347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.317474 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.317939 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:11.317948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:11.319362 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:11.320763 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.320769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.320772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.320977 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.321188 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:11.321189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:11.321189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:11.321290 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:11.321580 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:11.321587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:11.321744 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.321859 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.322009 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260810 14:58:11.322621 4082 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46563
I20260810 14:58:11.322705 4082 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:11.322844 4082 heartbeater.cc:511] Master 127.28.193.254:46563 requested a full tablet report, sending...
I20260810 14:58:11.323200 3408 ts_manager.cc:194] Registered new tserver with Master: bfe739550eda434cad631625ee1a697b (127.28.193.195:40125)
I20260810 14:58:11.323557 3408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:58369
I20260810 14:58:11.333256 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:58:11.339716 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.334553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.28.193.254:46563)
0810 14:58:11.334627 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.334629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.334674 (+ 45us) server_negotiation.cc:407] Connection header received
0810 14:58:11.334724 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.334725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.334738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.334789 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.335286 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.335748 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.336401 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.336621 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.336678 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.337699 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.337700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.337701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:11.337707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.338834 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.339124 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.339125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.339125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.339169 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:11.339290 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:11.339291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:11.339291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:11.339438 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.339516 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.339590 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.339650 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:58:11.341295 3408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:11.342886 3408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:11.349076 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:40125 (local address 127.0.0.1:58088)
0810 14:58:11.345373 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.345415 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.345499 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.345690 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.345693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.345696 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:11.345783 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.345786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.346809 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.346809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.347786 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.347790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.348774 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.348775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.348875 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.348889 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.348951 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.349000 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"spinlock_wait_cycles":1408,"thread_start_us":95,"threads_started":1}
I20260810 14:58:11.349404 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.28.193.195:40125)
0810 14:58:11.345493 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.345498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.345509 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:11.345580 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.345582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.345598 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.345677 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:11.345875 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.346719 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.347893 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.348630 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.348689 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.348742 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.349280 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.349324 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260810 14:58:11.349599 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:39409 (local address 127.0.0.1:40206)
0810 14:58:11.345460 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.345470 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.345526 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.346105 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.346106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.346108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:11.346149 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.346151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.347377 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.347378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.348344 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.348348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.349373 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.349374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.349435 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.349438 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.349494 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.349533 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":29,"thread_start_us":61,"threads_started":1}
I20260810 14:58:11.349704 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36853 (local address 127.0.0.1:41392)
0810 14:58:11.345605 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.345614 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.345650 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.346027 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.346028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.346030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:11.346080 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.346081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.347371 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.347372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.348531 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.348534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.349469 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.349470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.349536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.349540 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.349589 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.349629 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260810 14:58:11.349833 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.28.193.193:39409)
0810 14:58:11.345560 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.345565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.345576 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:11.345613 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.345615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.345629 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.345697 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:11.346217 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.347270 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.348439 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.349227 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.349290 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.349340 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.349697 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.349760 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
I20260810 14:58:11.349865 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.345363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.28.193.194:36853)
0810 14:58:11.345857 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.345862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.345871 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:11.345906 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.345909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.345924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.346018 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:11.346142 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.347262 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.348628 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.349308 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.349380 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.349432 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.349624 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.349778 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":80,"threads_started":1}
I20260810 14:58:11.350164 3990 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc066f37b5745b8ad654300b31944e6 (DEFAULT_TABLE table=test-table [id=1b8cdf11a1184174b04fd7d8a8502edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:11.350477 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc066f37b5745b8ad654300b31944e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:11.350469 3601 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc066f37b5745b8ad654300b31944e6 (DEFAULT_TABLE table=test-table [id=1b8cdf11a1184174b04fd7d8a8502edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:11.350471 3800 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc066f37b5745b8ad654300b31944e6 (DEFAULT_TABLE table=test-table [id=1b8cdf11a1184174b04fd7d8a8502edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:11.350793 3800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc066f37b5745b8ad654300b31944e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:11.350793 3601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc066f37b5745b8ad654300b31944e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:11.352867 4113 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Bootstrap starting.
I20260810 14:58:11.353551 4113 tablet_bootstrap.cc:654] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:11.354080 4114 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Bootstrap starting.
I20260810 14:58:11.354158 4113 log.cc:824] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:11.354372 4115 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Bootstrap starting.
I20260810 14:58:11.354700 4114 tablet_bootstrap.cc:654] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:11.354959 4115 tablet_bootstrap.cc:654] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:11.354996 4113 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: No bootstrap required, opened a new log
I20260810 14:58:11.355052 4113 ts_tablet_manager.cc:1397] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:58:11.355232 4114 log.cc:824] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:11.355558 4115 log.cc:824] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:11.355916 4114 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: No bootstrap required, opened a new log
I20260810 14:58:11.355988 4114 ts_tablet_manager.cc:1397] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:11.356333 4115 tablet_bootstrap.cc:492] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: No bootstrap required, opened a new log
I20260810 14:58:11.356405 4115 ts_tablet_manager.cc:1397] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:58:11.356961 4113 raft_consensus.cc:348] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.357136 4113 raft_consensus.cc:374] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:11.357173 4113 raft_consensus.cc:729] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe739550eda434cad631625ee1a697b, State: Initialized, Role: FOLLOWER
I20260810 14:58:11.357311 4113 consensus_queue.cc:260] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.357617 4082 heartbeater.cc:503] Master 127.28.193.254:46563 was elected leader, sending a full tablet report...
I20260810 14:58:11.357617 4114 raft_consensus.cc:348] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.357810 4114 raft_consensus.cc:374] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:11.357867 4114 raft_consensus.cc:729] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1947deb942a946cb9c4d44faf1c0e4cf, State: Initialized, Role: FOLLOWER
I20260810 14:58:11.357609 4113 ts_tablet_manager.cc:1428] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 14:58:11.358000 4114 consensus_queue.cc:260] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.358258 3702 heartbeater.cc:503] Master 127.28.193.254:46563 was elected leader, sending a full tablet report...
I20260810 14:58:11.358337 4115 raft_consensus.cc:348] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.358500 4115 raft_consensus.cc:374] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:11.358536 4115 raft_consensus.cc:729] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad2915a63d44babb65e1fd1a1e1d50d, State: Initialized, Role: FOLLOWER
I20260810 14:58:11.358565 4114 ts_tablet_manager.cc:1428] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:58:11.358656 4115 consensus_queue.cc:260] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.358937 4115 ts_tablet_manager.cc:1428] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 14:58:11.359167 3896 heartbeater.cc:503] Master 127.28.193.254:46563 was elected leader, sending a full tablet report...
W20260810 14:58:11.420701 3897 tablet.cc:2369] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:11.492830 4124 raft_consensus.cc:482] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:11.492954 4124 raft_consensus.cc:504] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.493311 4124 leader_election.cc:288] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1947deb942a946cb9c4d44faf1c0e4cf (127.28.193.193:39409), 9ad2915a63d44babb65e1fd1a1e1d50d (127.28.193.194:36853)
I20260810 14:58:11.496250 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.493575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:49667 (local address 127.28.193.194:36853)
0810 14:58:11.493627 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.493629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.493799 (+ 170us) server_negotiation.cc:407] Connection header received
0810 14:58:11.493851 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.493853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.493862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.493910 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:11.494074 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.494548 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.495557 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.495940 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.496003 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.496012 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.496148 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.496184 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:11.496259 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.493561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56379 (local address 127.28.193.193:39409)
0810 14:58:11.493612 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.493614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.493623 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:11.493653 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.493654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.493663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.493748 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:11.493898 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.494469 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.495461 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.495833 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.495897 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.495906 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.496155 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.496198 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:58:11.496259 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.493474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:39409 (local address 127.28.193.195:56379)
0810 14:58:11.493533 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.493544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.493590 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.493755 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.493757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.493759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:11.493837 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.493839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.494545 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.494546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.495382 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.495384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.495899 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.495900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.496051 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.496055 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.496134 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.496179 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:58:11.496269 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.493480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36853 (local address 127.28.193.195:49667)
0810 14:58:11.493731 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.493740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.493797 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.493914 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.493915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.493916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:11.494007 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.494009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.494615 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.494615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.495489 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:11.495492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.496011 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.496011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:11.496088 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.496092 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.496136 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.496176 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":62,"thread_start_us":120,"threads_started":1}
I20260810 14:58:11.496654 3630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc066f37b5745b8ad654300b31944e6" candidate_uuid: "bfe739550eda434cad631625ee1a697b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" is_pre_election: true
I20260810 14:58:11.496814 3630 raft_consensus.cc:2454] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe739550eda434cad631625ee1a697b in term 0.
I20260810 14:58:11.496881 3829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc066f37b5745b8ad654300b31944e6" candidate_uuid: "bfe739550eda434cad631625ee1a697b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" is_pre_election: true
I20260810 14:58:11.497033 3829 raft_consensus.cc:2454] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe739550eda434cad631625ee1a697b in term 0.
I20260810 14:58:11.497059 3925 leader_election.cc:302] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1947deb942a946cb9c4d44faf1c0e4cf, bfe739550eda434cad631625ee1a697b; no voters:
I20260810 14:58:11.497191 4124 raft_consensus.cc:2781] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:11.497236 4124 raft_consensus.cc:482] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:11.497257 4124 raft_consensus.cc:3037] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:11.498031 4124 raft_consensus.cc:504] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.498152 4124 leader_election.cc:288] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [CANDIDATE]: Term 1 election: Requested vote from peers 1947deb942a946cb9c4d44faf1c0e4cf (127.28.193.193:39409), 9ad2915a63d44babb65e1fd1a1e1d50d (127.28.193.194:36853)
I20260810 14:58:11.498317 3829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc066f37b5745b8ad654300b31944e6" candidate_uuid: "bfe739550eda434cad631625ee1a697b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d"
I20260810 14:58:11.498337 3630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc066f37b5745b8ad654300b31944e6" candidate_uuid: "bfe739550eda434cad631625ee1a697b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1947deb942a946cb9c4d44faf1c0e4cf"
I20260810 14:58:11.498392 3829 raft_consensus.cc:3037] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:11.498409 3630 raft_consensus.cc:3037] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:11.499109 3829 raft_consensus.cc:2454] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe739550eda434cad631625ee1a697b in term 1.
I20260810 14:58:11.499150 3630 raft_consensus.cc:2454] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe739550eda434cad631625ee1a697b in term 1.
I20260810 14:58:11.499287 3926 leader_election.cc:302] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad2915a63d44babb65e1fd1a1e1d50d, bfe739550eda434cad631625ee1a697b; no voters:
I20260810 14:58:11.499402 4124 raft_consensus.cc:2781] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:11.499572 4124 raft_consensus.cc:686] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 1 LEADER]: Becoming Leader. State: Replica: bfe739550eda434cad631625ee1a697b, State: Running, Role: LEADER
I20260810 14:58:11.499665 4124 consensus_queue.cc:237] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } }
I20260810 14:58:11.500499 3407 catalog_manager.cc:5495] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe739550eda434cad631625ee1a697b (127.28.193.195). New cstate: current_term: 1 leader_uuid: "bfe739550eda434cad631625ee1a697b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe739550eda434cad631625ee1a697b" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 } health_report { overall_health: UNKNOWN } } }
W20260810 14:58:11.519894 3703 tablet.cc:2369] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:11.532156 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.529922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.28.193.195:40125)
0810 14:58:11.529976 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.529978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.529986 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:11.530033 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.530034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.530052 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.530102 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:11.530215 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.530671 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.531411 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:11.531585 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:11.531626 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:11.531779 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:11.531909 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:11.531985 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.532041 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.532064 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:11.533865 3829 raft_consensus.cc:1261] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 1 FOLLOWER]: Refusing update from remote peer bfe739550eda434cad631625ee1a697b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:11.533874 3630 raft_consensus.cc:1261] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 1 FOLLOWER]: Refusing update from remote peer bfe739550eda434cad631625ee1a697b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:11.534169 4124 consensus_queue.cc:1035] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1947deb942a946cb9c4d44faf1c0e4cf" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 39409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:11.534293 4130 consensus_queue.cc:1035] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad2915a63d44babb65e1fd1a1e1d50d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:11.536368 4133 mvcc.cc:205] Tried to move back new op lower bound from 7316987459720159232 to 7316987459583238144. Current Snapshot: MvccSnapshot[applied={T|T < 7316987459720159232}]
I20260810 14:58:11.536739 4136 mvcc.cc:205] Tried to move back new op lower bound from 7316987459720159232 to 7316987459583238144. Current Snapshot: MvccSnapshot[applied={T|T < 7316987459720159232}]
I20260810 14:58:11.537038 4134 mvcc.cc:205] Tried to move back new op lower bound from 7316987459720159232 to 7316987459583238144. Current Snapshot: MvccSnapshot[applied={T|T < 7316987459720159232}]
I20260810 14:58:11.539248 3408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:11.539336 3408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:11.540004 3408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c43f88e10a594fb8a1565b36aa4f3733: Sending DeleteTablet for 3 replicas of tablet 7fc066f37b5745b8ad654300b31944e6
I20260810 14:58:11.540313 3990 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc066f37b5745b8ad654300b31944e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:11 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:58088
I20260810 14:58:11.540400 3601 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc066f37b5745b8ad654300b31944e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:11 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:40206
I20260810 14:58:11.540436 3800 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fc066f37b5745b8ad654300b31944e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:11 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41392
I20260810 14:58:11.540540 4146 tablet_replica.cc:330] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: stopping tablet replica
I20260810 14:58:11.540624 4146 raft_consensus.cc:2229] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:11.540740 4146 raft_consensus.cc:2258] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:11.540755 4149 tablet_replica.cc:330] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf: stopping tablet replica
I20260810 14:58:11.540932 4149 raft_consensus.cc:2229] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:11.540938 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 3508
I20260810 14:58:11.541088 4149 raft_consensus.cc:2258] T 7fc066f37b5745b8ad654300b31944e6 P 1947deb942a946cb9c4d44faf1c0e4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:11.541190 4146 ts_tablet_manager.cc:1905] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:11.543279 4146 ts_tablet_manager.cc:1918] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:11.543367 4146 log.cc:1196] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/7fc066f37b5745b8ad654300b31944e6
I20260810 14:58:11.543670 4146 ts_tablet_manager.cc:1939] T 7fc066f37b5745b8ad654300b31944e6 P bfe739550eda434cad631625ee1a697b: Deleting consensus metadata
I20260810 14:58:11.544117 3384 catalog_manager.cc:4841] TS bfe739550eda434cad631625ee1a697b (127.28.193.195:40125): tablet 7fc066f37b5745b8ad654300b31944e6 (table test-table [id=1b8cdf11a1184174b04fd7d8a8502edb]) successfully deleted
I20260810 14:58:11.550140 4150 tablet_replica.cc:330] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: stopping tablet replica
I20260810 14:58:11.550254 4150 raft_consensus.cc:2229] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:11.550367 4150 raft_consensus.cc:2258] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:11.550750 4150 ts_tablet_manager.cc:1905] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:11.551939 4150 ts_tablet_manager.cc:1918] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:11.552006 4150 log.cc:1196] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/7fc066f37b5745b8ad654300b31944e6
I20260810 14:58:11.552228 4150 ts_tablet_manager.cc:1939] T 7fc066f37b5745b8ad654300b31944e6 P 9ad2915a63d44babb65e1fd1a1e1d50d: Deleting consensus metadata
I20260810 14:58:11.552589 3386 catalog_manager.cc:4841] TS 9ad2915a63d44babb65e1fd1a1e1d50d (127.28.193.194:36853): tablet 7fc066f37b5745b8ad654300b31944e6 (table test-table [id=1b8cdf11a1184174b04fd7d8a8502edb]) successfully deleted
W20260810 14:58:11.554409 3385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:39409 (error 108)
I20260810 14:58:11.554627 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 3707
I20260810 14:58:11.562101 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.554813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:39409 (local address 127.0.0.1:40220)
0810 14:58:11.561999 (+ 7186us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.562061 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:39409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7155}
W20260810 14:58:11.583145 3385 catalog_manager.cc:4584] TS 1947deb942a946cb9c4d44faf1c0e4cf (127.28.193.193:39409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fc066f37b5745b8ad654300b31944e6: Network error: Client connection negotiation failed: client connection to 127.28.193.193:39409: connect: Connection refused (error 111)
I20260810 14:58:11.592129 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 3903
I20260810 14:58:11.603902 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 3367
2026-08-10T14:58:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2186 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T14:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:11Z Disabled control of system clock
I20260810 14:58:11.634344 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41599
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:41599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:58:11.714411 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:11.714613 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:11.714632 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:11.715709 4161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:11.715742 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:11.715754 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:11.715764 4161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:11.715778 4161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:11.717051 4161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:41599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:11.717244 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:11.717479 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:11.722174 4161 server_base.cc:1034] running on GCE node
W20260810 14:58:11.722136 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.722157 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.722147 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.722637 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:11.722937 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:11.724079 4161 hybrid_clock.cc:648] HybridClock initialized: now 1786373891724062 us; error 30 us; skew 500 ppm
I20260810 14:58:11.725256 4161 webserver.cc:460] Webserver started at http://127.28.193.254:37657/ using document root <none> and password file <none>
I20260810 14:58:11.725471 4161 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:11.725521 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:11.725646 4161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:11.726722 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "eb7c38d198db4f27b6068a15fe3f26ff"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b1c9331ddcafbb473064ef0cad01162"
server_key_iv: "b4c02daf177bc80be33a5e6c1ecd6568"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.727064 4161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "eb7c38d198db4f27b6068a15fe3f26ff"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b1c9331ddcafbb473064ef0cad01162"
server_key_iv: "b4c02daf177bc80be33a5e6c1ecd6568"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.728276 4161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:11.728987 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.729148 4161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:11.729241 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "eb7c38d198db4f27b6068a15fe3f26ff"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "9b1c9331ddcafbb473064ef0cad01162"
server_key_iv: "b4c02daf177bc80be33a5e6c1ecd6568"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.729305 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.732623 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.732862 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.732987 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.736639 4161 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:41599
I20260810 14:58:11.736668 4274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:41599 every 8 connection(s)
I20260810 14:58:11.736982 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:11.737620 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:11.738281 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 4161
I20260810 14:58:11.738375 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:11.738557 29447 external_mini_cluster.cc:1428] Setting key b136b91bf7e0d19e592c64dae0fa3b48
I20260810 14:58:11.741042 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.739063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.28.193.254:41599)
0810 14:58:11.739286 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.739295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.739329 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:58:11.739850 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.739860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.739918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.740068 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.740622 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.740627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.740627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:11.740631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.740679 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:11.740693 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.740766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.740805 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.740874 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260810 14:58:11.741086 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: Bootstrap starting.
I20260810 14:58:11.741806 4275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:11.742079 4275 log.cc:824] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:11.742677 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: No bootstrap required, opened a new log
I20260810 14:58:11.744216 4275 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } }
I20260810 14:58:11.744357 4275 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:11.744379 4275 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7c38d198db4f27b6068a15fe3f26ff, State: Initialized, Role: FOLLOWER
I20260810 14:58:11.744465 4275 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } }
I20260810 14:58:11.744524 4275 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:11.744547 4275 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:11.744576 4275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:11.745378 4275 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } }
I20260810 14:58:11.745471 4275 leader_election.cc:302] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7c38d198db4f27b6068a15fe3f26ff; no voters:
I20260810 14:58:11.745587 4275 leader_election.cc:288] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:11.745640 4281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:11.745817 4281 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [term 1 LEADER]: Becoming Leader. State: Replica: eb7c38d198db4f27b6068a15fe3f26ff, State: Running, Role: LEADER
I20260810 14:58:11.745851 4275 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:11.745934 4281 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } }
I20260810 14:58:11.746309 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7c38d198db4f27b6068a15fe3f26ff. Latest consensus state: current_term: 1 leader_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } } }
I20260810 14:58:11.746423 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:11.746307 4282 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7c38d198db4f27b6068a15fe3f26ff" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41599 } } }
I20260810 14:58:11.746511 4282 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:11.746636 4293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:11.747125 4293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:11.748481 4293 catalog_manager.cc:1347] Generated new cluster ID: 19a8e6ba2d6c401384218c7e55163841
I20260810 14:58:11.748533 4293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:11.772302 4293 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:11.772485 4293 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:11.775363 4293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: Generated new TSK 0
I20260810 14:58:11.775520 4293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:11.777325 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--builtin_ntp_servers=127.28.193.212:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:58:11.859509 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:11.859710 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:11.859732 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:11.860929 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:11.860971 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:11.860997 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:11.862720 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:11.862964 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:11.863260 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:11.868734 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.868778 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.868947 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.869074 4312 server_base.cc:1034] running on GCE node
I20260810 14:58:11.869277 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:11.869494 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:11.870630 4312 hybrid_clock.cc:648] HybridClock initialized: now 1786373891870616 us; error 30 us; skew 500 ppm
I20260810 14:58:11.871773 4312 webserver.cc:460] Webserver started at http://127.28.193.193:42867/ using document root <none> and password file <none>
I20260810 14:58:11.872015 4312 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:11.872061 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:11.872188 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:11.873255 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "437288d3f42e44358f42b814cf109b4b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "52d58ae036011d59db9bc5da1a714c73"
server_key_iv: "4e6cfbec44bdd05bb0b5dcaacfd19c1f"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.873605 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "437288d3f42e44358f42b814cf109b4b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "52d58ae036011d59db9bc5da1a714c73"
server_key_iv: "4e6cfbec44bdd05bb0b5dcaacfd19c1f"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.874828 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:11.875469 4331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.875602 4312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:11.875684 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "437288d3f42e44358f42b814cf109b4b"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "52d58ae036011d59db9bc5da1a714c73"
server_key_iv: "4e6cfbec44bdd05bb0b5dcaacfd19c1f"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.875753 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.879061 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.879313 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.879438 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.879628 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:11.879913 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:11.879945 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.879971 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:11.879994 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.885398 4312 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:34007
I20260810 14:58:11.885456 4500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:34007 every 8 connection(s)
I20260810 14:58:11.885769 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:11.888634 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.886461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33827 (local address 127.28.193.254:41599)
0810 14:58:11.886523 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.886525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.886727 (+ 202us) server_negotiation.cc:407] Connection header received
0810 14:58:11.887211 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:11.887212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:11.887219 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:11.887267 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:11.888393 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:11.888394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:11.888394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:11.888399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:11.888415 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:11.888419 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:11.888478 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:11.888548 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:58:11.888563 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:58:11.888815 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.886433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41599 (local address 127.28.193.193:33827)
0810 14:58:11.886618 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.886650 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:11.887115 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:11.887348 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:11.887352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:11.887362 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:11.888246 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:11.888247 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:11.888274 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:11.888303 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:11.888477 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:11.888482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:11.888485 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:11.888555 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 14:58:11.888616 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260810 14:58:11.889344 4502 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41599
I20260810 14:58:11.889423 4502 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:11.889544 4502 heartbeater.cc:511] Master 127.28.193.254:41599 requested a full tablet report, sending...
I20260810 14:58:11.890022 4207 ts_manager.cc:194] Registered new tserver with Master: 437288d3f42e44358f42b814cf109b4b (127.28.193.193:34007)
I20260810 14:58:11.892745 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 4312
I20260810 14:58:11.892829 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:11.892982 29447 external_mini_cluster.cc:1428] Setting key 78ffa0ca1c2b3773f1b1eff0305b6659
I20260810 14:58:11.893810 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--builtin_ntp_servers=127.28.193.212:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:58:11.974502 4509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:11.974704 4509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:11.974725 4509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:11.975867 4509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:11.975906 4509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:11.975929 4509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:11.977246 4509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:11.977473 4509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:11.977794 4509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:11.982689 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:11.982704 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.982736 4509 server_base.cc:1034] running on GCE node
W20260810 14:58:11.982704 4524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:11.983093 4509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:11.983285 4509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:11.984429 4509 hybrid_clock.cc:648] HybridClock initialized: now 1786373891984413 us; error 32 us; skew 500 ppm
I20260810 14:58:11.985587 4509 webserver.cc:460] Webserver started at http://127.28.193.194:41299/ using document root <none> and password file <none>
I20260810 14:58:11.985827 4509 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:11.985873 4509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:11.986002 4509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:11.987030 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "f9cfa87005164d5fb20d8d63f84dabcc"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0bf2a53913a54eb349de9182f5540d47"
server_key_iv: "691ac1f8d7026da7a9094c3cd3cfa7cb"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.987366 4509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "f9cfa87005164d5fb20d8d63f84dabcc"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0bf2a53913a54eb349de9182f5540d47"
server_key_iv: "691ac1f8d7026da7a9094c3cd3cfa7cb"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.988572 4509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:11.989261 4533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.989426 4509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:11.989514 4509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "f9cfa87005164d5fb20d8d63f84dabcc"
format_stamp: "Formatted at 2026-08-10 14:58:11 on dist-test-slave-cvwc"
server_key: "0bf2a53913a54eb349de9182f5540d47"
server_key_iv: "691ac1f8d7026da7a9094c3cd3cfa7cb"
server_key_version: "encryptionkey@0"
I20260810 14:58:11.989570 4509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:11.992729 4509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:11.992929 4509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:11.993022 4509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:11.993166 4509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:11.993412 4509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:11.993443 4509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.993469 4509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:11.993486 4509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:11.998703 4509 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:41303
I20260810 14:58:11.998775 4727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:41303 every 8 connection(s)
I20260810 14:58:11.999038 4509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:12.001784 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.999746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:55649 (local address 127.28.193.254:41599)
0810 14:58:11.999806 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:11.999808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:11.999990 (+ 182us) server_negotiation.cc:407] Connection header received
0810 14:58:12.000427 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.000428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.000434 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.000488 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.001530 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.001531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.001532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.001536 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.001552 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.001556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.001603 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.001662 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.001674 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:12.001940 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:11.999697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41599 (local address 127.28.193.194:55649)
0810 14:58:11.999874 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:11.999912 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.000349 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.000542 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.000545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.000553 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.001379 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.001379 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.001407 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.001436 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.001608 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.001610 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.001613 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.001664 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.001732 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260810 14:58:12.002379 4728 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41599
I20260810 14:58:12.002461 4728 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:12.002600 4728 heartbeater.cc:511] Master 127.28.193.254:41599 requested a full tablet report, sending...
I20260810 14:58:12.002912 4207 ts_manager.cc:194] Registered new tserver with Master: f9cfa87005164d5fb20d8d63f84dabcc (127.28.193.194:41303)
I20260810 14:58:12.008661 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 4509
I20260810 14:58:12.008749 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:12.008924 29447 external_mini_cluster.cc:1428] Setting key 21d88f13398f649963f4bba8df7e276d
I20260810 14:58:12.009924 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--builtin_ntp_servers=127.28.193.212:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:58:12.090667 4734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:12.090865 4734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:12.090885 4734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:12.092041 4734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:12.092080 4734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:12.092103 4734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:12.093487 4734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:12.093740 4734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:12.094022 4734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:12.098678 4741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.098716 4740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.098687 4744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:12.098883 4734 server_base.cc:1034] running on GCE node
I20260810 14:58:12.099073 4734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:12.099283 4734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:12.100441 4734 hybrid_clock.cc:648] HybridClock initialized: now 1786373892100431 us; error 30 us; skew 500 ppm
I20260810 14:58:12.101810 4734 webserver.cc:460] Webserver started at http://127.28.193.195:40765/ using document root <none> and password file <none>
I20260810 14:58:12.102067 4734 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:12.102113 4734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:12.102237 4734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:12.103257 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "8d65d3e2d5c042148d13a82af9713109"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "cf48783ecbeaf9775f778fd95b9cadf2"
server_key_iv: "4cc3eed570bec30e513112033e6a4605"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.103585 4734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "8d65d3e2d5c042148d13a82af9713109"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "cf48783ecbeaf9775f778fd95b9cadf2"
server_key_iv: "4cc3eed570bec30e513112033e6a4605"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.104785 4734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:12.105418 4754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.105554 4734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:12.105634 4734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "8d65d3e2d5c042148d13a82af9713109"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "cf48783ecbeaf9775f778fd95b9cadf2"
server_key_iv: "4cc3eed570bec30e513112033e6a4605"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.105701 4734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:12.108836 4734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:12.109088 4734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:12.109203 4734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:12.109382 4734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:12.109663 4734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:12.109694 4734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.109738 4734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:12.109766 4734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.115131 4734 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:41151
I20260810 14:58:12.115185 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:41151 every 8 connection(s)
I20260810 14:58:12.115468 4734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:12.118099 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.116105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:55253 (local address 127.28.193.254:41599)
0810 14:58:12.116152 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.116154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.116345 (+ 191us) server_negotiation.cc:407] Connection header received
0810 14:58:12.116753 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.116754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.116759 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.116806 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.117920 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.117921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.117921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.117925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.117939 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.117943 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.117993 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.118036 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.118047 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:58:12.118270 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.116078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41599 (local address 127.28.193.195:55253)
0810 14:58:12.116238 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.116269 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.116681 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.116876 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.116880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.116888 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.117783 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.117784 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.117812 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.117847 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.117992 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.117994 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.117996 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.118042 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.118087 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260810 14:58:12.118695 4935 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41599
I20260810 14:58:12.118778 4935 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:12.118921 4935 heartbeater.cc:511] Master 127.28.193.254:41599 requested a full tablet report, sending...
I20260810 14:58:12.119263 4207 ts_manager.cc:194] Registered new tserver with Master: 8d65d3e2d5c042148d13a82af9713109 (127.28.193.195:41151)
I20260810 14:58:12.125047 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 4734
I20260810 14:58:12.125156 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:12.125329 29447 external_mini_cluster.cc:1428] Setting key e5625214e1c0d35d755da5f371b687d8
I20260810 14:58:12.126142 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:58:12.128423 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.127439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.28.193.254:41599)
0810 14:58:12.127493 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.127495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.127590 (+ 95us) server_negotiation.cc:407] Connection header received
0810 14:58:12.127651 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.127652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.127656 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.127690 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.128265 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.128266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.128266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.128268 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.128278 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.128281 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.128307 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.128352 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.128361 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:58:12.129964 4207 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:12.131467 4207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:12.135155 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.28.193.193:34007)
0810 14:58:12.134127 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.134132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.134141 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:12.134167 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.134169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.134177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.134497 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.134856 (+ 359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.134862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.134863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.134866 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.134898 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.134916 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.135007 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.135080 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.135093 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 14:58:12.135159 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:34007 (local address 127.0.0.1:53050)
0810 14:58:12.133946 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.133954 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.133976 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.134273 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.134281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.134288 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.134747 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.134747 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.134761 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.134780 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.135004 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.135009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.135012 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.135076 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.135089 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":95,"threads_started":1}
I20260810 14:58:12.135394 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:41151 (local address 127.0.0.1:55114)
0810 14:58:12.133843 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.133851 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.133886 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.134407 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.134409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.134411 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.134876 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.134876 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.134881 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.134890 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.135263 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.135264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.135265 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.135321 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.135334 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":98,"threads_started":1}
I20260810 14:58:12.135638 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:41303 (local address 127.0.0.1:37790)
0810 14:58:12.133734 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.133775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.133835 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.134581 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.134582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.134584 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.135258 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.135259 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.135265 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.135276 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.135512 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.135513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.135514 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.135563 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.135576 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260810 14:58:12.135834 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.28.193.194:41303)
0810 14:58:12.134184 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.134188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.134198 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:12.134221 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.134223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.134231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.134301 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.135354 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.135364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.135364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.135368 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.135412 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.135438 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.135487 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.135664 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.135681 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":117,"threads_started":1}
I20260810 14:58:12.135947 4399 tablet_service.cc:1467] Processing CreateTablet for tablet a43c3727c32a4835890762cafb836a25 (DEFAULT_TABLE table=test-table [id=db82b486b975406f93fc78d146c31e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:12.136329 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43c3727c32a4835890762cafb836a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.136427 4602 tablet_service.cc:1467] Processing CreateTablet for tablet a43c3727c32a4835890762cafb836a25 (DEFAULT_TABLE table=test-table [id=db82b486b975406f93fc78d146c31e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:12.136675 4602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43c3727c32a4835890762cafb836a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.138828 4958 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: Bootstrap starting.
I20260810 14:58:12.138911 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.133808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.28.193.195:41151)
0810 14:58:12.133974 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.133979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.133990 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:12.134016 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.134018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.134028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.134104 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.134957 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.134965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.134965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.134970 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.135003 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.135027 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.135085 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.135493 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.135508 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260810 14:58:12.139436 4958 tablet_bootstrap.cc:654] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.139472 4833 tablet_service.cc:1467] Processing CreateTablet for tablet a43c3727c32a4835890762cafb836a25 (DEFAULT_TABLE table=test-table [id=db82b486b975406f93fc78d146c31e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:12.139735 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43c3727c32a4835890762cafb836a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.139740 4958 log.cc:824] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.140386 4958 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: No bootstrap required, opened a new log
I20260810 14:58:12.140447 4958 ts_tablet_manager.cc:1397] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:12.142050 4959 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Bootstrap starting.
I20260810 14:58:12.142091 4961 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Bootstrap starting.
I20260810 14:58:12.142617 4961 tablet_bootstrap.cc:654] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.142627 4958 raft_consensus.cc:348] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.142795 4959 tablet_bootstrap.cc:654] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.142797 4958 raft_consensus.cc:374] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.142834 4958 raft_consensus.cc:729] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437288d3f42e44358f42b814cf109b4b, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.142910 4961 log.cc:824] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.142989 4958 consensus_queue.cc:260] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.143095 4959 log.cc:824] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.143253 4958 ts_tablet_manager.cc:1428] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:58:12.143370 4502 heartbeater.cc:503] Master 127.28.193.254:41599 was elected leader, sending a full tablet report...
I20260810 14:58:12.143536 4961 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: No bootstrap required, opened a new log
I20260810 14:58:12.143589 4961 ts_tablet_manager.cc:1397] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:12.145676 4961 raft_consensus.cc:348] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.145848 4961 raft_consensus.cc:374] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.145872 4961 raft_consensus.cc:729] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d65d3e2d5c042148d13a82af9713109, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.145982 4961 consensus_queue.cc:260] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.146015 4959 tablet_bootstrap.cc:492] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: No bootstrap required, opened a new log
I20260810 14:58:12.146071 4959 ts_tablet_manager.cc:1397] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 14:58:12.146199 4961 ts_tablet_manager.cc:1428] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:58:12.146227 4935 heartbeater.cc:503] Master 127.28.193.254:41599 was elected leader, sending a full tablet report...
I20260810 14:58:12.148110 4959 raft_consensus.cc:348] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.148270 4959 raft_consensus.cc:374] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.148310 4959 raft_consensus.cc:729] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9cfa87005164d5fb20d8d63f84dabcc, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.148417 4959 consensus_queue.cc:260] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.148634 4959 ts_tablet_manager.cc:1428] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:58:12.148685 4728 heartbeater.cc:503] Master 127.28.193.254:41599 was elected leader, sending a full tablet report...
W20260810 14:58:12.249553 4730 tablet.cc:2369] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:12.280542 4964 raft_consensus.cc:482] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:12.280682 4964 raft_consensus.cc:504] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.281023 4964 leader_election.cc:288] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9cfa87005164d5fb20d8d63f84dabcc (127.28.193.194:41303), 8d65d3e2d5c042148d13a82af9713109 (127.28.193.195:41151)
I20260810 14:58:12.287767 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.281280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38835 (local address 127.28.193.195:41151)
0810 14:58:12.281337 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.281339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.281458 (+ 119us) server_negotiation.cc:407] Connection header received
0810 14:58:12.281515 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.281516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.281521 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.281575 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.287540 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.287542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.287542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.287547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.287564 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.287569 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.287615 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.287678 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.287692 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:58:12.287778 4507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.281209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:41303 (local address 127.28.193.193:45165)
0810 14:58:12.281283 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.281295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.281335 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.281476 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.281476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.281479 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.287459 (+ 5980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.287460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.287467 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.287479 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.287659 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.287660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.287661 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.287705 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.287717 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:58:12.287786 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.281286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:45165 (local address 127.28.193.194:41303)
0810 14:58:12.281331 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.281333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.281344 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:12.281396 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.281397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.281403 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.281461 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.287573 (+ 6112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.287574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.287574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.287578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.287595 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.287600 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.287658 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.287713 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.287725 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:12.287786 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.281225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:41151 (local address 127.28.193.193:38835)
0810 14:58:12.281387 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.281397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.281462 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.281590 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.281591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.281592 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:12.287420 (+ 5828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:12.287421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:12.287430 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:12.287445 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:12.287625 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:12.287626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:12.287626 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 14:58:12.287674 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:58:12.287687 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260810 14:58:12.288161 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43c3727c32a4835890762cafb836a25" candidate_uuid: "437288d3f42e44358f42b814cf109b4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d65d3e2d5c042148d13a82af9713109" is_pre_election: true
I20260810 14:58:12.288151 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43c3727c32a4835890762cafb836a25" candidate_uuid: "437288d3f42e44358f42b814cf109b4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" is_pre_election: true
I20260810 14:58:12.288313 4861 raft_consensus.cc:2454] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437288d3f42e44358f42b814cf109b4b in term 0.
I20260810 14:58:12.288313 4646 raft_consensus.cc:2454] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437288d3f42e44358f42b814cf109b4b in term 0.
I20260810 14:58:12.288539 4332 leader_election.cc:302] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437288d3f42e44358f42b814cf109b4b, f9cfa87005164d5fb20d8d63f84dabcc; no voters:
I20260810 14:58:12.288700 4964 raft_consensus.cc:2781] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:12.288758 4964 raft_consensus.cc:482] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:12.288774 4964 raft_consensus.cc:3037] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.289464 4964 raft_consensus.cc:504] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.289582 4964 leader_election.cc:288] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [CANDIDATE]: Term 1 election: Requested vote from peers f9cfa87005164d5fb20d8d63f84dabcc (127.28.193.194:41303), 8d65d3e2d5c042148d13a82af9713109 (127.28.193.195:41151)
I20260810 14:58:12.289739 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43c3727c32a4835890762cafb836a25" candidate_uuid: "437288d3f42e44358f42b814cf109b4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9cfa87005164d5fb20d8d63f84dabcc"
I20260810 14:58:12.289747 4861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43c3727c32a4835890762cafb836a25" candidate_uuid: "437288d3f42e44358f42b814cf109b4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d65d3e2d5c042148d13a82af9713109"
I20260810 14:58:12.289808 4646 raft_consensus.cc:3037] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.289829 4861 raft_consensus.cc:3037] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.290490 4646 raft_consensus.cc:2454] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437288d3f42e44358f42b814cf109b4b in term 1.
I20260810 14:58:12.290594 4861 raft_consensus.cc:2454] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437288d3f42e44358f42b814cf109b4b in term 1.
I20260810 14:58:12.290643 4332 leader_election.cc:302] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437288d3f42e44358f42b814cf109b4b, f9cfa87005164d5fb20d8d63f84dabcc; no voters:
I20260810 14:58:12.290729 4964 raft_consensus.cc:2781] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:12.290887 4964 raft_consensus.cc:686] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [term 1 LEADER]: Becoming Leader. State: Replica: 437288d3f42e44358f42b814cf109b4b, State: Running, Role: LEADER
I20260810 14:58:12.290979 4964 consensus_queue.cc:237] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } }
I20260810 14:58:12.291831 4206 catalog_manager.cc:5495] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 437288d3f42e44358f42b814cf109b4b (127.28.193.193). New cstate: current_term: 1 leader_uuid: "437288d3f42e44358f42b814cf109b4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437288d3f42e44358f42b814cf109b4b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 } health_report { overall_health: UNKNOWN } } }
I20260810 14:58:12.315526 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.314682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.28.193.193:34007)
0810 14:58:12.314746 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.314748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.314756 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:12.314791 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.314791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.314803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.314848 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.315319 (+ 471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.315320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.315320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.315331 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.315344 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.315348 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.315393 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.315441 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.315455 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:58:12.317202 4646 raft_consensus.cc:1261] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 1 FOLLOWER]: Refusing update from remote peer 437288d3f42e44358f42b814cf109b4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:12.317199 4861 raft_consensus.cc:1261] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 1 FOLLOWER]: Refusing update from remote peer 437288d3f42e44358f42b814cf109b4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:12.317502 4964 consensus_queue.cc:1035] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d65d3e2d5c042148d13a82af9713109" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 41151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:12.317647 4973 consensus_queue.cc:1035] T a43c3727c32a4835890762cafb836a25 P 437288d3f42e44358f42b814cf109b4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9cfa87005164d5fb20d8d63f84dabcc" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:12.319996 4978 mvcc.cc:205] Tried to move back new op lower bound from 7316987462928715776 to 7316987462824390656. Current Snapshot: MvccSnapshot[applied={T|T < 7316987462928715776}]
I20260810 14:58:12.320803 4979 mvcc.cc:205] Tried to move back new op lower bound from 7316987462928715776 to 7316987462824390656. Current Snapshot: MvccSnapshot[applied={T|T < 7316987462928715776 or (T in {7316987462928715776})}]
I20260810 14:58:12.322028 4206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:12.322125 4206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:12.322818 4206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7c38d198db4f27b6068a15fe3f26ff: Sending DeleteTablet for 3 replicas of tablet a43c3727c32a4835890762cafb836a25
I20260810 14:58:12.323128 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet a43c3727c32a4835890762cafb836a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:12 UTC) from {username='slave'} at 127.0.0.1:53050
I20260810 14:58:12.323148 4833 tablet_service.cc:1514] Processing DeleteTablet for tablet a43c3727c32a4835890762cafb836a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:12 UTC) from {username='slave'} at 127.0.0.1:55114
I20260810 14:58:12.323361 4991 tablet_replica.cc:330] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: stopping tablet replica
I20260810 14:58:12.323374 4602 tablet_service.cc:1514] Processing DeleteTablet for tablet a43c3727c32a4835890762cafb836a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:12 UTC) from {username='slave'} at 127.0.0.1:37790
I20260810 14:58:12.323465 4991 raft_consensus.cc:2229] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:12.323567 4991 raft_consensus.cc:2258] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:12.323915 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 4312
I20260810 14:58:12.323654 4992 tablet_replica.cc:330] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: stopping tablet replica
I20260810 14:58:12.329115 4992 raft_consensus.cc:2229] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:12.323983 4991 ts_tablet_manager.cc:1905] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:12.329239 4992 raft_consensus.cc:2258] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:12.329668 4992 ts_tablet_manager.cc:1905] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:12.330906 4991 ts_tablet_manager.cc:1918] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:12.330955 4992 ts_tablet_manager.cc:1918] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:12.330988 4991 log.cc:1196] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/a43c3727c32a4835890762cafb836a25
I20260810 14:58:12.331022 4992 log.cc:1196] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/a43c3727c32a4835890762cafb836a25
I20260810 14:58:12.331254 4992 ts_tablet_manager.cc:1939] T a43c3727c32a4835890762cafb836a25 P f9cfa87005164d5fb20d8d63f84dabcc: Deleting consensus metadata
I20260810 14:58:12.331254 4991 ts_tablet_manager.cc:1939] T a43c3727c32a4835890762cafb836a25 P 8d65d3e2d5c042148d13a82af9713109: Deleting consensus metadata
I20260810 14:58:12.331653 4179 catalog_manager.cc:4841] TS f9cfa87005164d5fb20d8d63f84dabcc (127.28.193.194:41303): tablet a43c3727c32a4835890762cafb836a25 (table test-table [id=db82b486b975406f93fc78d146c31e1f]) successfully deleted
I20260810 14:58:12.331691 4180 catalog_manager.cc:4841] TS 8d65d3e2d5c042148d13a82af9713109 (127.28.193.195:41151): tablet a43c3727c32a4835890762cafb836a25 (table test-table [id=db82b486b975406f93fc78d146c31e1f]) successfully deleted
W20260810 14:58:12.334698 4181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:34007 (error 108)
I20260810 14:58:12.335147 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.335040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:34007 (local address 127.0.0.1:53078)
0810 14:58:12.335084 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.335119 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:34007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 14:58:12.335271 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 4509
W20260810 14:58:12.335266 4181 catalog_manager.cc:4584] TS 437288d3f42e44358f42b814cf109b4b (127.28.193.193:34007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a43c3727c32a4835890762cafb836a25: Network error: Client connection negotiation failed: client connection to 127.28.193.193:34007: connect: Connection refused (error 111)
I20260810 14:58:12.351433 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 4734
I20260810 14:58:12.363747 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 4161
2026-08-10T14:58:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (758 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 14:58:12.385795 29447 mini_oidc.cc:152] Starting JWKS server
I20260810 14:58:12.387053 29447 webserver.cc:460] Webserver started at https://127.0.0.1:35241/ using document root <none> and password file <none>
I20260810 14:58:12.387148 29447 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:58:12.387313 29447 webserver.cc:460] Webserver started at http://127.0.0.1:46799/ using document root <none> and password file <none>
2026-08-10T14:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:12Z Disabled control of system clock
I20260810 14:58:12.395514 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36229
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:36229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35241/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/testchainca.cert with env {}
W20260810 14:58:12.476438 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:12.476667 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:12.476688 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:12.477896 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:12.477938 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:12.477949 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:12.477962 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:12.477984 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:58:12.478006 5003 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:58:12.478026 5003 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35241/jwks.json
I20260810 14:58:12.479693 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:36229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35241/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:12.479946 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:12.480152 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:12.485129 5022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.485185 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:12.485260 5003 server_base.cc:1034] running on GCE node
W20260810 14:58:12.485159 5016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:12.485646 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:12.485977 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:12.487107 5003 hybrid_clock.cc:648] HybridClock initialized: now 1786373892487097 us; error 27 us; skew 500 ppm
I20260810 14:58:12.488327 5003 webserver.cc:460] Webserver started at http://127.28.193.254:38251/ using document root <none> and password file <none>
I20260810 14:58:12.488564 5003 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:12.488611 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:12.488731 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:12.489864 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "351518b2e41ba3a0607f7d04fd5099db"
server_key_iv: "4bd3a4c5f6479083543f813ab63f3762"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.490197 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "351518b2e41ba3a0607f7d04fd5099db"
server_key_iv: "4bd3a4c5f6479083543f813ab63f3762"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.491477 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:12.492154 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.492326 5003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.492417 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "351518b2e41ba3a0607f7d04fd5099db"
server_key_iv: "4bd3a4c5f6479083543f813ab63f3762"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.492476 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:12.509959 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:12.510252 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:12.510360 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:12.514101 5003 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:36229
I20260810 14:58:12.514135 5108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:36229 every 8 connection(s)
I20260810 14:58:12.514521 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:12.515529 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.517798 5110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc: Bootstrap starting.
I20260810 14:58:12.518340 5110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.518586 5110 log.cc:824] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.519299 5110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc: No bootstrap required, opened a new log
I20260810 14:58:12.519936 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5003
I20260810 14:58:12.520040 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:12.520224 29447 external_mini_cluster.cc:1428] Setting key 1f3f3298ce31898a4a55572ed77ab3f1
I20260810 14:58:12.521235 5110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } }
I20260810 14:58:12.521389 5110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.521426 5110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5ec812ff5245829c9fd8c56e3b2cfc, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.521559 5110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } }
I20260810 14:58:12.521643 5110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:12.521669 5110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:12.521737 5110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.522249 5110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } }
I20260810 14:58:12.522334 5110 leader_election.cc:302] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5ec812ff5245829c9fd8c56e3b2cfc; no voters:
I20260810 14:58:12.522480 5110 leader_election.cc:288] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:12.522509 5118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:12.522621 5118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [term 1 LEADER]: Becoming Leader. State: Replica: 5c5ec812ff5245829c9fd8c56e3b2cfc, State: Running, Role: LEADER
I20260810 14:58:12.522702 5118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } }
I20260810 14:58:12.522742 5110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:12.523196 5121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } } }
I20260810 14:58:12.523198 5122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c5ec812ff5245829c9fd8c56e3b2cfc. Latest consensus state: current_term: 1 leader_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5ec812ff5245829c9fd8c56e3b2cfc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36229 } } }
I20260810 14:58:12.523299 5121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.523303 5122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.523520 5130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:12.523972 5130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:12.525451 5130 catalog_manager.cc:1347] Generated new cluster ID: aea924f24f8e41659a5f30b038cc661a
I20260810 14:58:12.525501 5130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:12.526440 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.520737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.28.193.254:36229)
0810 14:58:12.520916 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.520924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.520948 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:58:12.521195 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.521201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.521255 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.521376 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.521612 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.523130 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.524395 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.524912 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.525026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.525700 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.525704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.525704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.525921 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.525964 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.525975 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.526031 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.526115 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.526272 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260810 14:58:12.533988 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:58:12.537473 5130 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:12.538017 5130 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:12.540306 5130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c5ec812ff5245829c9fd8c56e3b2cfc: Generated new TSK 0
I20260810 14:58:12.540467 5130 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:58:12.567516 5157 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 14:58:12.567696 5157 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:58:12.567836 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.551784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.28.193.254:36229)
0810 14:58:12.551842 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.551844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.551933 (+ 89us) server_negotiation.cc:407] Connection header received
0810 14:58:12.552020 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.552021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.552033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.552089 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:58:12.552233 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.552718 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.553814 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.554020 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.554074 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.554285 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:58:12.567572 (+ 13287us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:58:12.567760 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 14:58:12.567895 5116 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33240: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 14:58:12.568545 5155 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:12.551667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36229 (local address 127.0.0.1:33240)
0810 14:58:12.551840 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.551851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.551925 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.552104 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.552105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.552107 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:58:12.552179 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:12.552181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.552791 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.552791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:12.553739 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:12.553743 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.554098 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.554099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:12.554163 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.554187 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:58:12.567873 (+ 13686us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:58:12.568085 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.193.254:36229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.193.254:36229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35241/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 14:58:12.568992 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5003
2026-08-10T14:58:12Z chronyd exiting
I20260810 14:58:12.617585 29447 test_util.cc:175] -----------------------------------------------
I20260810 14:58:12.617671 29447 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786373869351853-29447-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 14:58:12.618536 29447 mini_oidc.cc:152] Starting JWKS server
I20260810 14:58:12.619197 29447 webserver.cc:460] Webserver started at https://127.0.0.1:41075/ using document root <none> and password file <none>
I20260810 14:58:12.619240 29447 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:58:12.619383 29447 webserver.cc:460] Webserver started at http://127.0.0.1:41775/ using document root <none> and password file <none>
2026-08-10T14:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:12Z Disabled control of system clock
I20260810 14:58:12.627736 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46723
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:46723
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41075/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/testchainca.cert with env {}
W20260810 14:58:12.709905 5169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:12.710101 5169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:12.710121 5169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:12.711256 5169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:12.711295 5169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:12.711306 5169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:12.711318 5169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:12.711330 5169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:58:12.711354 5169 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:58:12.711369 5169 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41075/jwks.json
I20260810 14:58:12.712908 5169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:46723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41075/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:12.713145 5169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:12.713353 5169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:12.717986 5187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.718011 5182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.718010 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:12.718180 5169 server_base.cc:1034] running on GCE node
I20260810 14:58:12.718448 5169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:12.718745 5169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:12.719895 5169 hybrid_clock.cc:648] HybridClock initialized: now 1786373892719879 us; error 36 us; skew 500 ppm
I20260810 14:58:12.721135 5169 webserver.cc:460] Webserver started at http://127.28.193.254:39629/ using document root <none> and password file <none>
I20260810 14:58:12.721369 5169 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:12.721417 5169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:12.721556 5169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:12.722640 5169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "85da2f7216b24f51b07574018555645c"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "407ef36df502bda05c829b5127369844"
server_key_iv: "709f1141a26d4ffd26a36f62c91393d7"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.722930 5169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "85da2f7216b24f51b07574018555645c"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "407ef36df502bda05c829b5127369844"
server_key_iv: "709f1141a26d4ffd26a36f62c91393d7"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.724076 5169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:12.724692 5197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.724834 5169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:12.724918 5169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "85da2f7216b24f51b07574018555645c"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "407ef36df502bda05c829b5127369844"
server_key_iv: "709f1141a26d4ffd26a36f62c91393d7"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.724987 5169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:12.745118 5169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:12.745461 5169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:12.745597 5169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:12.749271 5169 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:46723
I20260810 14:58:12.749315 5282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:46723 every 8 connection(s)
I20260810 14:58:12.749655 5169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:12.750355 5283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.752038 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5169
I20260810 14:58:12.752136 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:12.752315 29447 external_mini_cluster.cc:1428] Setting key 6a54d947df28978a76a8b17b0d1cb26e
I20260810 14:58:12.753473 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c: Bootstrap starting.
I20260810 14:58:12.754249 5283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.754556 5283 log.cc:824] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.755324 5283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c: No bootstrap required, opened a new log
I20260810 14:58:12.757401 5283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } }
I20260810 14:58:12.757565 5283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.757613 5283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85da2f7216b24f51b07574018555645c, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.757767 5283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } }
I20260810 14:58:12.757867 5283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:12.757910 5283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:12.757961 5283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.758409 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.752832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.28.193.254:46723)
0810 14:58:12.753052 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.753062 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.753099 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:58:12.753629 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.753639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.753742 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.753902 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.754062 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.755319 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.756132 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.756811 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.756959 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.757518 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.757522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.757522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.757756 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.757800 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.757815 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.757878 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.758090 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.758174 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260810 14:58:12.758651 5283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } }
I20260810 14:58:12.758759 5283 leader_election.cc:302] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85da2f7216b24f51b07574018555645c; no voters:
I20260810 14:58:12.758936 5283 leader_election.cc:288] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:12.759020 5290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:12.759171 5290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [term 1 LEADER]: Becoming Leader. State: Replica: 85da2f7216b24f51b07574018555645c, State: Running, Role: LEADER
I20260810 14:58:12.759197 5283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:12.759258 5290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } }
I20260810 14:58:12.759337 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:58:12.759625 5290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85da2f7216b24f51b07574018555645c. Latest consensus state: current_term: 1 leader_uuid: "85da2f7216b24f51b07574018555645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } } }
I20260810 14:58:12.759706 5290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.759799 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85da2f7216b24f51b07574018555645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85da2f7216b24f51b07574018555645c" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46723 } } }
I20260810 14:58:12.759918 5299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:12.759932 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.760452 5299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:12.761848 5299 catalog_manager.cc:1347] Generated new cluster ID: 93b73922aef24882880919b9d9038e08
I20260810 14:58:12.761916 5299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:12.765702 5299 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:12.766399 5299 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:12.774708 5299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85da2f7216b24f51b07574018555645c: Generated new TSK 0
I20260810 14:58:12.774885 5299 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:58:12.800117 5330 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 14:58:12.800213 5330 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:58:12.800559 5287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.780153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.28.193.254:46723)
0810 14:58:12.780209 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.780211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.780303 (+ 92us) server_negotiation.cc:407] Connection header received
0810 14:58:12.780380 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.780380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.780393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.780457 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:58:12.780575 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.781061 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.781860 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.782081 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.782138 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.782302 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:58:12.800326 (+ 18024us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:58:12.800488 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48912: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 14:58:12.800618 5287 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48912: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 14:58:12.800717 5329 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:12.780102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46723 (local address 127.0.0.1:48912)
0810 14:58:12.780247 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:12.780256 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:12.780312 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:12.780461 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:12.780462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:12.780464 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:58:12.780521 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:12.780523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.781153 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.781154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:12.781785 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:12.781788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.782147 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.782148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:12.782208 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.782214 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:58:12.800458 (+ 18244us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:58:12.800611 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.193.254:46723: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41075/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260810 14:58:12.801038 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5169
2026-08-10T14:58:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 14:58:12.847909 29447 mini_oidc.cc:152] Starting JWKS server
I20260810 14:58:12.848836 29447 webserver.cc:460] Webserver started at https://127.0.0.1:46645/ using document root <none> and password file <none>
I20260810 14:58:12.848906 29447 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:58:12.849085 29447 webserver.cc:460] Webserver started at http://127.0.0.1:42467/ using document root <none> and password file <none>
2026-08-10T14:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:12Z Disabled control of system clock
I20260810 14:58:12.859056 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34061
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:58:12.940699 5338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:12.940925 5338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:12.940948 5338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:12.942171 5338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:12.942214 5338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:12.942225 5338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:12.942236 5338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:12.942246 5338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:58:12.942260 5338 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:58:12.942283 5338 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46645/jwks.json
I20260810 14:58:12.943710 5338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:12.944007 5338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:12.944293 5338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:12.948966 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.948968 5343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:12.948982 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:12.949165 5338 server_base.cc:1034] running on GCE node
I20260810 14:58:12.949429 5338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:12.949687 5338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:12.950850 5338 hybrid_clock.cc:648] HybridClock initialized: now 1786373892950835 us; error 29 us; skew 500 ppm
I20260810 14:58:12.952044 5338 webserver.cc:460] Webserver started at http://127.28.193.254:43727/ using document root <none> and password file <none>
I20260810 14:58:12.952256 5338 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:12.952296 5338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:12.952394 5338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:12.953449 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "6f2bcd531a5a4992a685f77d992c0791"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "496105f0d008969770e391a0fe78c16a"
server_key_iv: "2e10ade1e3216a78e2891020d451eeab"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.953791 5338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "6f2bcd531a5a4992a685f77d992c0791"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "496105f0d008969770e391a0fe78c16a"
server_key_iv: "2e10ade1e3216a78e2891020d451eeab"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.954993 5338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:12.955669 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:12.955835 5338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:12.955927 5338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "6f2bcd531a5a4992a685f77d992c0791"
format_stamp: "Formatted at 2026-08-10 14:58:12 on dist-test-slave-cvwc"
server_key: "496105f0d008969770e391a0fe78c16a"
server_key_iv: "2e10ade1e3216a78e2891020d451eeab"
server_key_version: "encryptionkey@0"
I20260810 14:58:12.956002 5338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:12.975771 5338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:12.976066 5338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:12.976176 5338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:12.979885 5338 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34061
I20260810 14:58:12.979936 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34061 every 8 connection(s)
I20260810 14:58:12.980266 5338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:12.980973 5435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:12.983366 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5338
I20260810 14:58:12.983457 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:12.983623 29447 external_mini_cluster.cc:1428] Setting key 634b2fdafa22bcbd5ac9bb8ad452eb40
I20260810 14:58:12.983747 5435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: Bootstrap starting.
I20260810 14:58:12.984453 5435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:12.984727 5435 log.cc:824] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:12.985394 5435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: No bootstrap required, opened a new log
I20260810 14:58:12.987142 5435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } }
I20260810 14:58:12.987293 5435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:12.987329 5435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2bcd531a5a4992a685f77d992c0791, State: Initialized, Role: FOLLOWER
I20260810 14:58:12.987458 5435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } }
I20260810 14:58:12.987550 5435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:12.987581 5435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:12.987629 5435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:12.988263 5435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } }
I20260810 14:58:12.988363 5435 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2bcd531a5a4992a685f77d992c0791; no voters:
I20260810 14:58:12.988512 5435 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:12.988546 5444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:12.988658 5444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [term 1 LEADER]: Becoming Leader. State: Replica: 6f2bcd531a5a4992a685f77d992c0791, State: Running, Role: LEADER
I20260810 14:58:12.988742 5444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } }
I20260810 14:58:12.988787 5435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:12.988914 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:12.984125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.28.193.254:34061)
0810 14:58:12.984323 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:12.984331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:12.984357 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:58:12.984831 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:12.984841 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:12.984910 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:12.985048 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:12.985257 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.986206 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.986956 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:12.987368 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:12.987493 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:12.988148 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:12.988152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:12.988152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:12.988316 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:12.988358 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:12.988374 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:12.988439 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:12.988511 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:58:12.988681 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260810 14:58:12.989070 5444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2bcd531a5a4992a685f77d992c0791. Latest consensus state: current_term: 1 leader_uuid: "6f2bcd531a5a4992a685f77d992c0791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } } }
I20260810 14:58:12.989144 5444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.989239 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2bcd531a5a4992a685f77d992c0791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2bcd531a5a4992a685f77d992c0791" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34061 } } }
I20260810 14:58:12.989353 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:12.989881 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 14:58:12.990517 5465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:12.990563 5465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:12.990605 5452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:12.990749 5452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:12.992113 5452 catalog_manager.cc:1347] Generated new cluster ID: e79d6c4afdf643489c2cd83326c87c23
I20260810 14:58:12.992165 5452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:13.010259 5452 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:13.011029 5452 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:13.013988 5452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2bcd531a5a4992a685f77d992c0791: Generated new TSK 0
I20260810 14:58:13.014174 5452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:13.024402 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:13.020786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.28.193.254:34061)
0810 14:58:13.020854 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:13.020856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:13.020867 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:13.020937 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:13.020938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:13.020951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:13.021028 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:58:13.021129 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.021943 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.022663 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.022989 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.023056 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:13.023232 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:58:13.024190 (+ 958us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:58:13.024334 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:58:13.024461 5441 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 14:58:13.026279 5477 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:13.020672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34061 (local address 127.0.0.1:59062)
0810 14:58:13.020816 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:13.020826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:13.020882 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:13.021026 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:13.021027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:13.021029 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:58:13.021088 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:13.021090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.022011 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.022011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:13.022596 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:13.022597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.023079 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.023081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:13.023138 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:13.023143 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:58:13.024290 (+ 1147us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:58:13.026185 (+ 1895us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.193.254:34061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1716,"thread_start_us":80,"threads_started":1}
I20260810 14:58:13.026695 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5338
E20260810 14:58:13.038614 5478 webserver.cc:533] Webserver: sslize failed:
E20260810 14:58:13.038689 5478 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T14:58:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T14:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:13Z Disabled control of system clock
I20260810 14:58:13.078524 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:39621
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:39621
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/testchainca.cert with env {}
W20260810 14:58:13.161219 5482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:13.161427 5482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:13.161448 5482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:13.162575 5482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:13.162609 5482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:13.162621 5482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:13.162631 5482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:13.162642 5482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:58:13.162653 5482 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:58:13.162662 5482 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 14:58:13.164135 5482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:39621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:39621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:13.164333 5482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:13.164497 5482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:13.169149 5491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:13.169166 5495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:13.169164 5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:13.169330 5482 server_base.cc:1034] running on GCE node
I20260810 14:58:13.169620 5482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:13.169945 5482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:13.171099 5482 hybrid_clock.cc:648] HybridClock initialized: now 1786373893171081 us; error 26 us; skew 500 ppm
I20260810 14:58:13.172359 5482 webserver.cc:460] Webserver started at http://127.28.193.254:38085/ using document root <none> and password file <none>
I20260810 14:58:13.172608 5482 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:13.172657 5482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:13.172787 5482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:13.173894 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "dfe47f1119604ab492a87f2e25011814"
format_stamp: "Formatted at 2026-08-10 14:58:13 on dist-test-slave-cvwc"
server_key: "cf6c156f0aaf21064c66aea0eb3d65fa"
server_key_iv: "9c91cd63e04dbac450106e74f9f6bbc9"
server_key_version: "encryptionkey@0"
I20260810 14:58:13.174235 5482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "dfe47f1119604ab492a87f2e25011814"
format_stamp: "Formatted at 2026-08-10 14:58:13 on dist-test-slave-cvwc"
server_key: "cf6c156f0aaf21064c66aea0eb3d65fa"
server_key_iv: "9c91cd63e04dbac450106e74f9f6bbc9"
server_key_version: "encryptionkey@0"
I20260810 14:58:13.175514 5482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:13.176303 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:13.176503 5482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:13.176595 5482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "dfe47f1119604ab492a87f2e25011814"
format_stamp: "Formatted at 2026-08-10 14:58:13 on dist-test-slave-cvwc"
server_key: "cf6c156f0aaf21064c66aea0eb3d65fa"
server_key_iv: "9c91cd63e04dbac450106e74f9f6bbc9"
server_key_version: "encryptionkey@0"
I20260810 14:58:13.176659 5482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:13.198277 5482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:13.198565 5482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:13.198671 5482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:13.202977 5482 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:39621
I20260810 14:58:13.203034 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:39621 every 8 connection(s)
I20260810 14:58:13.203429 5482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:13.204638 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:13.208046 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814: Bootstrap starting.
I20260810 14:58:13.208894 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:13.209266 5592 log.cc:824] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:13.210134 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814: No bootstrap required, opened a new log
I20260810 14:58:13.211987 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } }
I20260810 14:58:13.212131 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:13.212162 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe47f1119604ab492a87f2e25011814, State: Initialized, Role: FOLLOWER
I20260810 14:58:13.212297 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } }
I20260810 14:58:13.212363 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:13.212395 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:13.212440 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:13.212958 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } }
I20260810 14:58:13.213022 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5482
I20260810 14:58:13.213057 5592 leader_election.cc:302] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfe47f1119604ab492a87f2e25011814; no voters:
I20260810 14:58:13.213120 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:13.213213 5592 leader_election.cc:288] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:13.213279 5596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:13.213299 29447 external_mini_cluster.cc:1428] Setting key e5463f4520850b2c664c848ac1174fd0
I20260810 14:58:13.213478 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:13.213832 5596 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [term 1 LEADER]: Becoming Leader. State: Replica: dfe47f1119604ab492a87f2e25011814, State: Running, Role: LEADER
I20260810 14:58:13.213935 5596 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } }
I20260810 14:58:13.214540 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfe47f1119604ab492a87f2e25011814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } } }
I20260810 14:58:13.214634 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:13.215035 5624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:13.215142 5601 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfe47f1119604ab492a87f2e25011814. Latest consensus state: current_term: 1 leader_uuid: "dfe47f1119604ab492a87f2e25011814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfe47f1119604ab492a87f2e25011814" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39621 } } }
I20260810 14:58:13.215199 5601 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:13.215539 5624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:13.217464 5624 catalog_manager.cc:1347] Generated new cluster ID: db6337bcbed146d59df39aa9538dc98c
I20260810 14:58:13.217509 5624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:13.219197 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:13.213772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.28.193.254:39621)
0810 14:58:13.214150 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:13.214161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:13.214193 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:58:13.214320 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:13.214327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:13.214397 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:13.214799 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:13.214903 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.216240 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.217035 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.217666 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.217833 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:13.218489 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:13.218494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:13.218494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:13.218709 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:13.218750 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:13.218765 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:13.218834 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:13.218893 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:58:13.218981 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":70,"threads_started":1}
I20260810 14:58:13.220062 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:58:13.224793 5624 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:13.225244 5624 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:13.227162 5624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfe47f1119604ab492a87f2e25011814: Generated new TSK 0
I20260810 14:58:13.227310 5624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:13.291518 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:13.236191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.28.193.254:39621)
0810 14:58:13.236235 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:13.236237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:13.236328 (+ 91us) server_negotiation.cc:407] Connection header received
0810 14:58:13.236382 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:13.236383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:13.236407 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:13.236486 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:58:13.236583 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.237349 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.238154 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.238493 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.238565 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:13.238746 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:58:13.291188 (+ 52442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:58:13.291431 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47292: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 14:58:13.291570 5646 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:13.236116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:39621 (local address 127.0.0.1:47292)
0810 14:58:13.236266 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:13.236276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:13.236330 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:13.236484 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:13.236484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:13.236486 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:58:13.236533 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:13.236535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.237437 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.237437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:13.238084 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:13.238086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:13.238595 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:13.238596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:13.238646 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:13.238652 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:58:13.291361 (+ 52709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:58:13.291491 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.193.254:39621: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
W20260810 14:58:13.291611 5602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47292: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260810 14:58:13.291894 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5482
2026-08-10T14:58:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (248 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:13 dist-test-slave-cvwc krb5kdc[5664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:13 dist-test-slave-cvwc krb5kdc[5664](info): set up 2 sockets
Aug 10 14:58:13 dist-test-slave-cvwc krb5kdc[5664](info): commencing operation
krb5kdc: starting...
W20260810 14:58:14.640668 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:14 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:14.758788 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36207
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:36207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:14.841467 5686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:14.841681 5686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:14.841744 5686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:14.842885 5686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:14.842931 5686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:14.842954 5686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:14.842974 5686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:14.842998 5686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:14.844497 5686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:36207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:14.844760 5686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:14.844969 5686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:14.850980 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:14.850997 5697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:14.850980 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:14.851243 5686 server_base.cc:1034] running on GCE node
I20260810 14:58:14.851608 5686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:14.851931 5686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:14.853096 5686 hybrid_clock.cc:648] HybridClock initialized: now 1786373894853080 us; error 32 us; skew 500 ppm
Aug 10 14:58:14 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:14.856166 5686 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:14.856637 5686 webserver.cc:460] Webserver started at http://127.28.193.254:44729/ using document root <none> and password file <none>
I20260810 14:58:14.856901 5686 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:14.856945 5686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:14.857069 5686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:14.858408 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "336d5b9658ca46a7b2c743ed6da0d32b"
format_stamp: "Formatted at 2026-08-10 14:58:14 on dist-test-slave-cvwc"
server_key: "93f80d644f5c928767bceb3bd0055eb7"
server_key_iv: "8527ad0de08778aabb85681393faad37"
server_key_version: "encryptionkey@0"
I20260810 14:58:14.858835 5686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "336d5b9658ca46a7b2c743ed6da0d32b"
format_stamp: "Formatted at 2026-08-10 14:58:14 on dist-test-slave-cvwc"
server_key: "93f80d644f5c928767bceb3bd0055eb7"
server_key_iv: "8527ad0de08778aabb85681393faad37"
server_key_version: "encryptionkey@0"
I20260810 14:58:14.860499 5686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 14:58:14.861321 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:14.861477 5686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:14.861568 5686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "336d5b9658ca46a7b2c743ed6da0d32b"
format_stamp: "Formatted at 2026-08-10 14:58:14 on dist-test-slave-cvwc"
server_key: "93f80d644f5c928767bceb3bd0055eb7"
server_key_iv: "8527ad0de08778aabb85681393faad37"
server_key_version: "encryptionkey@0"
I20260810 14:58:14.861640 5686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:14.874624 5686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:14.877411 5686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:14.877537 5686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:14.881341 5686 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:36207
I20260810 14:58:14.881388 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:36207 every 8 connection(s)
I20260810 14:58:14.881702 5686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:14.882688 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:14.884004 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5686
I20260810 14:58:14.884104 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:14.884311 29447 external_mini_cluster.cc:1428] Setting key b9d2274e6576b8ad4d96c111fa2f749d
I20260810 14:58:14.885283 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b: Bootstrap starting.
I20260810 14:58:14.886001 5804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:14.886282 5804 log.cc:824] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:14.886917 5804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b: No bootstrap required, opened a new log
I20260810 14:58:14.888702 5804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } }
I20260810 14:58:14.888849 5804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:14.888882 5804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 336d5b9658ca46a7b2c743ed6da0d32b, State: Initialized, Role: FOLLOWER
I20260810 14:58:14.889010 5804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } }
I20260810 14:58:14.889093 5804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:14.889123 5804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:14.889171 5804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:14.889668 5804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } }
I20260810 14:58:14.889797 5804 leader_election.cc:302] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 336d5b9658ca46a7b2c743ed6da0d32b; no voters:
I20260810 14:58:14.889948 5804 leader_election.cc:288] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:14.889969 5810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:14.890072 5810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [term 1 LEADER]: Becoming Leader. State: Replica: 336d5b9658ca46a7b2c743ed6da0d32b, State: Running, Role: LEADER
I20260810 14:58:14.890153 5810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } }
I20260810 14:58:14.890200 5804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:14.890473 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 336d5b9658ca46a7b2c743ed6da0d32b. Latest consensus state: current_term: 1 leader_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } } }
I20260810 14:58:14.890543 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:14.890704 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "336d5b9658ca46a7b2c743ed6da0d32b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36207 } } }
Aug 10 14:58:14 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:14.890767 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:14.890888 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:14.891373 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:14.893186 5819 catalog_manager.cc:1347] Generated new cluster ID: cd932e53da07410da9952f17b58df3a1
I20260810 14:58:14.893231 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:14.895319 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:14.884864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.28.193.254:36207)
0810 14:58:14.885083 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:14.885092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:14.885124 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:58:14.885661 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:14.885672 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:14.885759 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:14.885900 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:14.886554 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:14.887503 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:14.888265 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:14.888663 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:14.888760 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:14.891319 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:14.891323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:14.891323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:14.891335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:14.893081 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:14.894088 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:14.894091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:14.894091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:14.894191 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:14.894368 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:14.894368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:14.894369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:14.894644 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:14.894757 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:14.894991 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:58:14.895119 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":77,"threads_started":1}
I20260810 14:58:14.904532 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:14.905241 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:14.909269 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 336d5b9658ca46a7b2c743ed6da0d32b: Generated new TSK 0
I20260810 14:58:14.909439 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:14.936332 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:15.018157 5843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:15.018357 5843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:15.018378 5843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:15.019512 5843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:15.019552 5843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:15.019579 5843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:15.021059 5843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:15.021287 5843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:15.021503 5843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:15.026275 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:15.026388 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:15.026280 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:15.026291 5843 server_base.cc:1034] running on GCE node
I20260810 14:58:15.026674 5843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:15.026899 5843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:15.028043 5843 hybrid_clock.cc:648] HybridClock initialized: now 1786373895028030 us; error 29 us; skew 500 ppm
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.030510 5843 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:15.030929 5843 webserver.cc:460] Webserver started at http://127.28.193.193:34441/ using document root <none> and password file <none>
I20260810 14:58:15.031157 5843 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:15.031198 5843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:15.031288 5843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:15.032250 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "fbf498bf83924652a8500d64a9877b5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "bbba7c1e797afe92e2728d67fdb39e25"
server_key_iv: "6ee1f82f00c9ebce7aa77dff0517489b"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.032553 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "fbf498bf83924652a8500d64a9877b5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "bbba7c1e797afe92e2728d67fdb39e25"
server_key_iv: "6ee1f82f00c9ebce7aa77dff0517489b"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.033804 5843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:15.034502 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.034632 5843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:15.034708 5843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "fbf498bf83924652a8500d64a9877b5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "bbba7c1e797afe92e2728d67fdb39e25"
server_key_iv: "6ee1f82f00c9ebce7aa77dff0517489b"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.034773 5843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:15.062809 5843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:15.064945 5843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:15.065083 5843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:15.065307 5843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:15.065610 5843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:15.065644 5843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.065677 5843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:15.065699 5843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.071195 5843 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:42001
I20260810 14:58:15.071259 6036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:42001 every 8 connection(s)
I20260810 14:58:15.071550 5843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:15.071800 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 5843
I20260810 14:58:15.071913 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:15.072124 29447 external_mini_cluster.cc:1428] Setting key 919056345350d4b8c858a74dd799b40f
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 14:58:15.081413 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.072297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:59413 (local address 127.28.193.254:36207)
0810 14:58:15.072360 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.072362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.072574 (+ 212us) server_negotiation.cc:407] Connection header received
0810 14:58:15.073488 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.073489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.073501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.073571 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.074190 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.074763 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.075941 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.076233 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.076304 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.078590 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.078592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.078592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.078599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.079965 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.080434 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.080435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.080435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.080492 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.080766 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.080766 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.080767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.080890 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.080942 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.081268 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.081344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:58:15.081573 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.072298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36207 (local address 127.28.193.193:59413)
0810 14:58:15.072448 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:15.072489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:15.073381 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:15.073629 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:15.073632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:15.073922 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:15.074113 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.074118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.074847 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.074848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.075854 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.075857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.076308 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.076308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.076406 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.076866 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:15.076874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:15.078499 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:15.080053 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.080060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.080063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.080346 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.080551 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.080552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.080553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.080698 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.080959 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:15.080966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:15.081132 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 14:58:15.081253 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 14:58:15.081381 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260810 14:58:15.082085 6038 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36207
I20260810 14:58:15.082181 6038 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:15.082366 6038 heartbeater.cc:511] Master 127.28.193.254:36207 requested a full tablet report, sending...
I20260810 14:58:15.082854 5740 ts_manager.cc:194] Registered new tserver with Master: fbf498bf83924652a8500d64a9877b5b (127.28.193.193:42001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
I20260810 14:58:15.083527 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:59413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:15.096478 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:15.177434 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:15.177644 6047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:15.177666 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:15.178818 6047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:15.178860 6047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:15.178884 6047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:15.180408 6047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:15.180636 6047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:15.180837 6047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:15.185570 6060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:15.185562 6061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:15.185688 6047 server_base.cc:1034] running on GCE node
W20260810 14:58:15.185604 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:15.186024 6047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:15.186240 6047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:15.187389 6047 hybrid_clock.cc:648] HybridClock initialized: now 1786373895187380 us; error 29 us; skew 500 ppm
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.189852 6047 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:15.190274 6047 webserver.cc:460] Webserver started at http://127.28.193.194:40469/ using document root <none> and password file <none>
I20260810 14:58:15.190466 6047 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:15.190549 6047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:15.190681 6047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:15.191797 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "31eb6ff0e5fb4cca93bda783277fc910"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "00709ca73395133fd18de9f5924e7b84"
server_key_iv: "cbc87177d8d856568979f324bb5a1fd4"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.192090 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "31eb6ff0e5fb4cca93bda783277fc910"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "00709ca73395133fd18de9f5924e7b84"
server_key_iv: "cbc87177d8d856568979f324bb5a1fd4"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.193298 6047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:15.194001 6073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.194142 6047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:15.194217 6047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "31eb6ff0e5fb4cca93bda783277fc910"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "00709ca73395133fd18de9f5924e7b84"
server_key_iv: "cbc87177d8d856568979f324bb5a1fd4"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.194283 6047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:15.211607 6047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:15.213764 6047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:15.213896 6047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:15.214128 6047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:15.214459 6047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:15.214496 6047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.214526 6047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:15.214550 6047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.220026 6047 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:42863
I20260810 14:58:15.220081 6260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:42863 every 8 connection(s)
I20260810 14:58:15.220372 6047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:15.221271 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 6047
I20260810 14:58:15.221387 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:15.221594 29447 external_mini_cluster.cc:1428] Setting key 2a5ab68d19bf3915fba7c3dfb86451ae
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:15.230089 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.221072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:56807 (local address 127.28.193.254:36207)
0810 14:58:15.221148 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.221150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.221388 (+ 238us) server_negotiation.cc:407] Connection header received
0810 14:58:15.222607 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.222608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.222618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.222654 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.223424 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.223907 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.224877 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.225074 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.225130 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.227504 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.227506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.227506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.227513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.228748 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.229199 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.229200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.229200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.229244 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.229480 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.229480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.229481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.229573 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.229618 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.229953 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.230022 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:58:15.230350 6265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.221037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36207 (local address 127.28.193.194:56807)
0810 14:58:15.221224 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:15.221289 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:15.222508 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:15.222727 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:15.222731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:15.223073 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:15.223343 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.223350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.224008 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.224009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.224795 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.224798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.225144 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.225145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.225263 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.225732 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:15.225743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:15.227415 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:15.228825 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.228832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.228835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.229131 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.229298 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.229298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.229299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.229425 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.229626 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:15.229631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:15.229814 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 14:58:15.229941 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 14:58:15.230108 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 14:58:15.230836 6261 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36207
I20260810 14:58:15.230928 6261 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:15.231086 6261 heartbeater.cc:511] Master 127.28.193.254:36207 requested a full tablet report, sending...
I20260810 14:58:15.231431 5740 ts_manager.cc:194] Registered new tserver with Master: 31eb6ff0e5fb4cca93bda783277fc910 (127.28.193.194:42863)
I20260810 14:58:15.231799 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:56807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:15.246384 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:15.326813 6268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:15.327008 6268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:15.327029 6268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:15.328130 6268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:15.328169 6268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:15.328192 6268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:15.329624 6268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:15.329874 6268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:15.330073 6268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:15.334815 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:15.334800 6278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:15.334821 6276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:15.335054 6268 server_base.cc:1034] running on GCE node
I20260810 14:58:15.335263 6268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:15.335505 6268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:15.336653 6268 hybrid_clock.cc:648] HybridClock initialized: now 1786373895336637 us; error 30 us; skew 500 ppm
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.339157 6268 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:15.339598 6268 webserver.cc:460] Webserver started at http://127.28.193.195:42379/ using document root <none> and password file <none>
I20260810 14:58:15.339773 6268 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:15.339812 6268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:15.339910 6268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:15.340921 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "1d3d0afcef8c4ba696f377332bef7a5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "954b3584a89051b8686cb662e3385aaf"
server_key_iv: "52219a0b3fe3c26b08bfbc6d7b03cac6"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.341225 6268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "1d3d0afcef8c4ba696f377332bef7a5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "954b3584a89051b8686cb662e3385aaf"
server_key_iv: "52219a0b3fe3c26b08bfbc6d7b03cac6"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.342491 6268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:15.343178 6290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.343345 6268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:15.343426 6268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "1d3d0afcef8c4ba696f377332bef7a5b"
format_stamp: "Formatted at 2026-08-10 14:58:15 on dist-test-slave-cvwc"
server_key: "954b3584a89051b8686cb662e3385aaf"
server_key_iv: "52219a0b3fe3c26b08bfbc6d7b03cac6"
server_key_version: "encryptionkey@0"
I20260810 14:58:15.343492 6268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:15.358193 6268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:15.360159 6268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:15.360267 6268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:15.360445 6268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:15.360749 6268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:15.360783 6268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.360805 6268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:15.360827 6268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:15.366366 6268 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:33517
I20260810 14:58:15.366411 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:33517 every 8 connection(s)
I20260810 14:58:15.366715 6268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:15.371220 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 6268
I20260810 14:58:15.371313 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:15.371511 29447 external_mini_cluster.cc:1428] Setting key bf611fae82ba7b9242469c48c9127085
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:15.375411 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.367368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40781 (local address 127.28.193.254:36207)
0810 14:58:15.367448 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.367451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.367598 (+ 147us) server_negotiation.cc:407] Connection header received
0810 14:58:15.368505 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.368506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.368516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.368566 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.369130 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.369629 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.370574 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.370760 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.370809 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.372905 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.372906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.372906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.372913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.374216 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.374610 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.374610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.374611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.374657 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.374885 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.374886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.374886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.374982 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.375027 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.375292 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.375347 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:58:15.375607 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.367338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36207 (local address 127.28.193.195:40781)
0810 14:58:15.367484 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:15.367522 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:15.368422 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:15.368623 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:15.368627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:15.368879 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:15.369066 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.369070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.369694 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.369694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.370507 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:15.370510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.370816 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.370816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:15.370915 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.371412 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:15.371423 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:15.372822 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:15.374297 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.374304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.374306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.374544 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.374706 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:15.374706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:15.374706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:15.374823 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:15.375057 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:15.375063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:15.375183 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 14:58:15.375280 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:58:15.375399 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260810 14:58:15.376021 6478 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36207
I20260810 14:58:15.376102 6478 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:15.376240 6478 heartbeater.cc:511] Master 127.28.193.254:36207 requested a full tablet report, sending...
I20260810 14:58:15.376569 5740 ts_manager.cc:194] Registered new tserver with Master: 1d3d0afcef8c4ba696f377332bef7a5b (127.28.193.195:33517)
I20260810 14:58:15.376909 5740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:40781
I20260810 14:58:15.383183 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/test-admin.keytab.
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.398005 29447 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 14:58:15.399580 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:15.398827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.28.193.193:42001)
0810 14:58:15.398994 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.398999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.399007 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:15.399032 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.399034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.399047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.399110 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.399511 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60606: BlockingRecv error: recv got EOF from 127.0.0.1:60606 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
W20260810 14:58:15.399645 29447 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.400493 29447 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.401247 29447 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:58:15.407886 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.401706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.28.193.193:42001)
0810 14:58:15.401757 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.401758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.401808 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:58:15.401849 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.401849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.401858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.401901 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.402206 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.402903 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.403551 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.403858 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.403910 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.405646 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.405654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.405654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.405660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.407027 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.407320 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.407323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.407324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.407379 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.407514 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.407514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.407514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.407626 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.407724 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.407758 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.407816 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 14:58:15.409447 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:15.408990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.28.193.193:42001)
0810 14:58:15.409036 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.409038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.409047 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:15.409111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.409112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.409121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.409174 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.409404 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60622: BlockingRecv error: recv got EOF from 127.0.0.1:60622 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 14:58:15.409523 29447 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.410414 29447 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.411166 29447 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:58:15.417080 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.411611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.28.193.193:42001)
0810 14:58:15.411655 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.411657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.411684 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:58:15.411748 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.411749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.411758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.411800 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.412121 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.412583 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.413214 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.413396 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.413448 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.415196 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.415197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.415198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.415204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.416183 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.416461 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.416462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.416463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.416509 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.416667 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.416667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.416668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.416750 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.416825 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.416925 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.417001 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 14:58:15.418380 6493 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:15.417921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.28.193.193:42001)
0810 14:58:15.417972 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.417974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.417995 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:58:15.418059 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.418060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.418068 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.418111 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.418342 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60632: BlockingRecv error: recv got EOF from 127.0.0.1:60632 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 14:58:15.418434 29447 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786373869351853-29447-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.419224 29447 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:15.419951 29447 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[5664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 14:58:15.425354 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:15.420368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.28.193.193:42001)
0810 14:58:15.420417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:15.420419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:15.420427 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:15.420476 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:15.420476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:15.420483 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:15.420516 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:15.420786 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.421209 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.421820 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:15.421995 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:15.422039 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:15.423667 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:15.423668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:15.423668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:15.423674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:15.424641 (+ 967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.424898 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.424899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.424899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.424937 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:15.425052 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:15.425052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:15.425052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:15.425121 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:15.425175 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:15.425238 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 14:58:15.425285 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:15.425902 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5843
I20260810 14:58:15.440605 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 6047
I20260810 14:58:15.465886 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 6268
I20260810 14:58:15.479446 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 5686
2026-08-10T14:58:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2008 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[6522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[6522](info): set up 2 sockets
Aug 10 14:58:15 dist-test-slave-cvwc krb5kdc[6522](info): commencing operation
krb5kdc: starting...
W20260810 14:58:16.654392 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:16 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab.
Entry for principal oryx/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab.
I20260810 14:58:16.770840 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44547
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab
--principal=oryx/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:16.859558 6547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:16.859804 6547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:16.859836 6547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:16.861054 6547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:16.861099 6547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:16.861117 6547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:16.861133 6547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:16.861148 6547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:16.862787 6547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab
--principal=oryx/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:16.863050 6547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:16.863298 6547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:16.872706 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:16.872709 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:16.872709 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:16.873358 6547 server_base.cc:1034] running on GCE node
I20260810 14:58:16.873656 6547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:16.874063 6547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:16.875233 6547 hybrid_clock.cc:648] HybridClock initialized: now 1786373896875209 us; error 40 us; skew 500 ppm
Aug 10 14:58:16 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:16.878618 6547 init.cc:377] Logged in from keytab as oryx/127.28.193.254@KRBTEST.COM (short username oryx)
I20260810 14:58:16.879175 6547 webserver.cc:460] Webserver started at http://127.28.193.254:36687/ using document root <none> and password file <none>
I20260810 14:58:16.879400 6547 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:16.879465 6547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:16.879607 6547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:16.880843 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "139086d48a2f416e870b433ec4c427cc"
format_stamp: "Formatted at 2026-08-10 14:58:16 on dist-test-slave-cvwc"
server_key: "f47db051d3e101e877549707accacaa1"
server_key_iv: "c428e8095995e6fb89d950c3ed4689f9"
server_key_version: "encryptionkey@0"
I20260810 14:58:16.881273 6547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "139086d48a2f416e870b433ec4c427cc"
format_stamp: "Formatted at 2026-08-10 14:58:16 on dist-test-slave-cvwc"
server_key: "f47db051d3e101e877549707accacaa1"
server_key_iv: "c428e8095995e6fb89d950c3ed4689f9"
server_key_version: "encryptionkey@0"
I20260810 14:58:16.882794 6547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:16.883594 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:16.883759 6547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:16.883843 6547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "139086d48a2f416e870b433ec4c427cc"
format_stamp: "Formatted at 2026-08-10 14:58:16 on dist-test-slave-cvwc"
server_key: "f47db051d3e101e877549707accacaa1"
server_key_iv: "c428e8095995e6fb89d950c3ed4689f9"
server_key_version: "encryptionkey@0"
I20260810 14:58:16.883919 6547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:16.895673 6547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:16.897748 6547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:16.897886 6547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:16.901647 6547 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:44547
I20260810 14:58:16.901695 6654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:44547 every 8 connection(s)
I20260810 14:58:16.902077 6547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:16.903035 6655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:16.905355 6655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Bootstrap starting.
I20260810 14:58:16.905961 6655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:16.906219 6655 log.cc:824] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:16.906860 6655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: No bootstrap required, opened a new log
I20260810 14:58:16.908423 6655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } }
I20260810 14:58:16.908560 6655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:16.908592 6655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139086d48a2f416e870b433ec4c427cc, State: Initialized, Role: FOLLOWER
I20260810 14:58:16.908785 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 6547
I20260810 14:58:16.908780 6655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } }
I20260810 14:58:16.908861 6655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:16.908891 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:16.908900 6655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:16.908942 6655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:16.909097 29447 external_mini_cluster.cc:1428] Setting key de579a7bf9cb2bc25d7ebd2d86e0e08b
I20260810 14:58:16.909917 6655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } }
I20260810 14:58:16.910009 6655 leader_election.cc:302] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139086d48a2f416e870b433ec4c427cc; no voters:
I20260810 14:58:16.910151 6655 leader_election.cc:288] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:16.910267 6661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:16.910404 6655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:16.910436 6661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 LEADER]: Becoming Leader. State: Replica: 139086d48a2f416e870b433ec4c427cc, State: Running, Role: LEADER
I20260810 14:58:16.910533 6661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } }
I20260810 14:58:16.910925 6661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139086d48a2f416e870b433ec4c427cc. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } }
I20260810 14:58:16.911019 6661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:16.910933 6663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } }
I20260810 14:58:16.911118 6663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:16.911947 6680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:16.912622 6680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:16.914285 6680 catalog_manager.cc:1347] Generated new cluster ID: 217a14952eb94220ba367531acc804f5
I20260810 14:58:16.914336 6680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:16 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:16.924077 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:16.909638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.28.193.254:44547)
0810 14:58:16.909848 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:16.909859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:16.909894 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:58:16.910010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:16.910018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:16.910078 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:16.910238 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:16.911032 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:16.912480 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:16.913469 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:16.913932 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:16.914039 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:16.920390 (+ 6351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:16.920394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:16.920395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:16.920409 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:16.922400 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:16.922833 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:16.922837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:16.922837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:16.922940 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:16.923134 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:16.923135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:16.923135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:16.923423 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:16.923558 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:16.923684 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 14:58:16.923829 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":123,"threads_started":1}
I20260810 14:58:16.925765 6680 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:16.926446 6680 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:16.936636 6680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Generated new TSK 0
I20260810 14:58:16.936852 6680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
I20260810 14:58:16.965914 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:17.046741 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:17.046950 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:17.046972 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:17.048034 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:17.048066 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:17.048090 6700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:17.049475 6700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:17.049649 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:17.049854 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.054605 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.054651 6708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.054615 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.054821 6700 server_base.cc:1034] running on GCE node
I20260810 14:58:17.055006 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.055238 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.056378 6700 hybrid_clock.cc:648] HybridClock initialized: now 1786373897056367 us; error 31 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.058861 6700 init.cc:377] Logged in from keytab as oryx/127.28.193.193@KRBTEST.COM (short username oryx)
I20260810 14:58:17.059280 6700 webserver.cc:460] Webserver started at http://127.28.193.193:33893/ using document root <none> and password file <none>
I20260810 14:58:17.059492 6700 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.059542 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.059665 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:17.060700 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "d1cb14a973b591a2a31566324e6a49f7"
server_key_iv: "9633dd32c0eddb8d6d96f3817ab9317b"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.061034 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "d1cb14a973b591a2a31566324e6a49f7"
server_key_iv: "9633dd32c0eddb8d6d96f3817ab9317b"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.062397 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:17.063172 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.063330 6700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.063418 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "d1cb14a973b591a2a31566324e6a49f7"
server_key_iv: "9633dd32c0eddb8d6d96f3817ab9317b"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.063479 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.073386 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.075140 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:17.075263 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.075455 6700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:17.075721 6700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:17.075755 6700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.075781 6700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:17.075805 6700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.081333 6700 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45807
I20260810 14:58:17.081465 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45807 every 8 connection(s)
I20260810 14:58:17.081694 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:17.090267 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.082453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:55869 (local address 127.28.193.254:44547)
0810 14:58:17.082507 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.082509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.082662 (+ 153us) server_negotiation.cc:407] Connection header received
0810 14:58:17.083175 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.083176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.083190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.083243 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.084133 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.084625 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.085560 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.085801 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.085862 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.087898 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.087899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.087899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.087909 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.089118 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.089491 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.089491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.089492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.089538 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.089742 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.089743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.089744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.089857 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.089907 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.090128 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.090200 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:17.090438 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.082388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:55869)
0810 14:58:17.082554 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.082595 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.083090 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.083308 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.083311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.083879 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.084061 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.084066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.084687 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.084687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.085475 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.085478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.085867 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.085868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.085964 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.086468 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.086476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.087811 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.089196 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.089202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.089204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.089432 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.089581 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.089582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.089582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.089671 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.089907 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.089911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.090026 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.090109 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.090224 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260810 14:58:17.090910 6901 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:17.090934 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 6700
I20260810 14:58:17.091005 6901 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:17.091032 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:17.091172 6901 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:17.091212 29447 external_mini_cluster.cc:1428] Setting key fbe13e83599fbb88893f4c18644063dd
I20260810 14:58:17.091696 6602 ts_manager.cc:194] Registered new tserver with Master: fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807)
I20260810 14:58:17.092442 6602 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:55869
WARNING: no policy specified for oryx/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
I20260810 14:58:17.114063 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:17.194460 6907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:17.194648 6907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:17.194669 6907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:17.195724 6907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:17.195756 6907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:17.195780 6907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:17.197160 6907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:17.197342 6907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:17.197513 6907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.202308 6912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.202332 6919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.202333 6913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.202487 6907 server_base.cc:1034] running on GCE node
I20260810 14:58:17.202678 6907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.202936 6907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.204080 6907 hybrid_clock.cc:648] HybridClock initialized: now 1786373897204068 us; error 30 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.206506 6907 init.cc:377] Logged in from keytab as oryx/127.28.193.194@KRBTEST.COM (short username oryx)
I20260810 14:58:17.206955 6907 webserver.cc:460] Webserver started at http://127.28.193.194:41357/ using document root <none> and password file <none>
I20260810 14:58:17.207183 6907 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.207231 6907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.207358 6907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:17.208367 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "605ef25bc3ea4c84855d440bbfd6903d"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "2c16ed735c4b8ed065d2893a7447067f"
server_key_iv: "0e8e559b5df9b474180eab1f12d2d99d"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.208698 6907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "605ef25bc3ea4c84855d440bbfd6903d"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "2c16ed735c4b8ed065d2893a7447067f"
server_key_iv: "0e8e559b5df9b474180eab1f12d2d99d"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.209985 6907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 14:58:17.210721 6931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.210873 6907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.210956 6907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "605ef25bc3ea4c84855d440bbfd6903d"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "2c16ed735c4b8ed065d2893a7447067f"
server_key_iv: "0e8e559b5df9b474180eab1f12d2d99d"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.211016 6907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.232276 6907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.234303 6907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:17.234443 6907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.234656 6907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:17.234963 6907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:17.234997 6907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.235026 6907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:17.235049 6907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.240509 6907 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:46199
I20260810 14:58:17.240549 7105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:46199 every 8 connection(s)
I20260810 14:58:17.240875 6907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:17.248908 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 6907
I20260810 14:58:17.249028 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:17.249238 29447 external_mini_cluster.cc:1428] Setting key 063cc7597661a4fa4ff8a3105e6d2c55
I20260810 14:58:17.249325 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.241635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:34811 (local address 127.28.193.254:44547)
0810 14:58:17.241702 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.241704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.241915 (+ 211us) server_negotiation.cc:407] Connection header received
0810 14:58:17.242418 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.242419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.242430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.242492 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.243504 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.244026 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.244968 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.245155 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.245200 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.247136 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.247137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.247138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.247144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.248188 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.248513 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.248514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.248514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.248555 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.248729 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.248730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.248730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.248856 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.248924 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.249152 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.249235 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:58:17.249533 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.241589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:34811)
0810 14:58:17.241790 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.241832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.242333 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.242559 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.242563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.243137 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.243403 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.243412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.244103 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.244104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.244903 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.244905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.245207 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.245208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.245299 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.245785 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.245794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.247051 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.248250 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.248255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.248256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.248458 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.248596 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.248596 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.248597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.248680 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.248911 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.248914 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.249021 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.249124 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.249241 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260810 14:58:17.250008 7106 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:17.250100 7106 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:17.250265 7106 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:17.250674 6601 ts_manager.cc:194] Registered new tserver with Master: 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
I20260810 14:58:17.251044 6601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:34811
WARNING: no policy specified for oryx/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
I20260810 14:58:17.274222 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:17.356153 7117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:17.356346 7117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:17.356369 7117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:17.357489 7117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:17.357530 7117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:17.357554 7117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:17.359011 7117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:17.359220 7117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:17.359436 7117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.364202 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.364202 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.364229 7126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.364488 7117 server_base.cc:1034] running on GCE node
I20260810 14:58:17.364693 7117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.364957 7117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.366111 7117 hybrid_clock.cc:648] HybridClock initialized: now 1786373897366097 us; error 29 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.371254 7117 init.cc:377] Logged in from keytab as oryx/127.28.193.195@KRBTEST.COM (short username oryx)
I20260810 14:58:17.371675 7117 webserver.cc:460] Webserver started at http://127.28.193.195:34875/ using document root <none> and password file <none>
I20260810 14:58:17.371884 7117 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.371925 7117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.372013 7117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:17.373015 7117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "6b1442114b594142b9112d03e7bdbc7c"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "459b9fab6b3fe32749b08cfa479d166b"
server_key_iv: "a7a2383f3436f0677a69b8e4088a2cf8"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.373318 7117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "6b1442114b594142b9112d03e7bdbc7c"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "459b9fab6b3fe32749b08cfa479d166b"
server_key_iv: "a7a2383f3436f0677a69b8e4088a2cf8"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.374650 7117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:17.375372 7144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.375543 7117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.375625 7117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "6b1442114b594142b9112d03e7bdbc7c"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "459b9fab6b3fe32749b08cfa479d166b"
server_key_iv: "a7a2383f3436f0677a69b8e4088a2cf8"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.375681 7117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.385121 7117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.386855 7117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:17.386965 7117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.387169 7117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:17.387442 7117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:17.387483 7117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.387509 7117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:17.387526 7117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.393548 7117 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:42045
I20260810 14:58:17.393592 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:42045 every 8 connection(s)
I20260810 14:58:17.393901 7117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:17.398933 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 7117
I20260810 14:58:17.399057 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:17.399243 29447 external_mini_cluster.cc:1428] Setting key 6fb1b5814115c90d639aa6d06db73c41
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:17.402562 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.394592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:59923 (local address 127.28.193.254:44547)
0810 14:58:17.394639 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.394641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.394817 (+ 176us) server_negotiation.cc:407] Connection header received
0810 14:58:17.395312 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.395313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.395326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.395376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.396274 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.396767 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.397699 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.397911 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.397974 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.400150 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.400151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.400151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.400159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.401461 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.401824 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.401825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.401825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.401896 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.402093 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.402094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.402094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.402195 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.402242 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.402434 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.402498 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:58:17.402750 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.394571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:59923)
0810 14:58:17.394710 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.394745 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.395218 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.395444 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.395455 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.396001 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.396180 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.396184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.396847 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.396848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.397617 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.397619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.397982 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.397982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.398095 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.398498 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.398505 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.400072 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.401524 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.401529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.401530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.401758 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.401955 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.401955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.401955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.402048 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.402236 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.402239 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.402347 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.402421 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.402545 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260810 14:58:17.403177 7329 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:17.403266 7329 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:17.403441 7329 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:17.403789 6601 ts_manager.cc:194] Registered new tserver with Master: 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:17.404148 6601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:59923
I20260810 14:58:17.411073 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.253.keytab.
Entry for principal oryx/127.28.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.253.keytab.
I20260810 14:58:17.436305 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:44487
--webserver_interface=127.28.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.253.keytab
--principal=oryx/127.28.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44547,127.28.193.253:44487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:17.517656 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:17.517895 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:17.517925 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:17.519065 7336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:17.519109 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:17.519127 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:17.519143 7336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:17.519158 7336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:17.520628 7336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44547,127.28.193.253:44487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.253.keytab
--principal=oryx/127.28.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:44487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
W20260810 14:58:17.520751 7336 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.193.254:44547,127.28.193.253:44487'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 14:58:17.520906 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:17.521112 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.525777 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.525765 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.525856 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.525915 7336 server_base.cc:1034] running on GCE node
I20260810 14:58:17.526194 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.526484 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.527621 7336 hybrid_clock.cc:648] HybridClock initialized: now 1786373897527610 us; error 26 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.530097 7336 init.cc:377] Logged in from keytab as oryx/127.28.193.253@KRBTEST.COM (short username oryx)
I20260810 14:58:17.530462 7336 webserver.cc:460] Webserver started at http://127.28.193.253:39477/ using document root <none> and password file <none>
I20260810 14:58:17.530701 7336 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.530746 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.530862 7336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:17.531859 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/instance:
uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "f2d6265a58b561b9f849bb8dbe07c784"
server_key_iv: "ad0f391e7b4f10cff90ff335ac16a7e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.532194 7336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal/instance:
uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "f2d6265a58b561b9f849bb8dbe07c784"
server_key_iv: "ad0f391e7b4f10cff90ff335ac16a7e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.533442 7336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:17.534289 7357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.534446 7336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.534530 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "f2d6265a58b561b9f849bb8dbe07c784"
server_key_iv: "ad0f391e7b4f10cff90ff335ac16a7e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.534584 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.548869 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.550961 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:17.551072 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.554790 7336 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.253:44487
I20260810 14:58:17.554834 7428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.253:44487 every 8 connection(s)
I20260810 14:58:17.555155 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
I20260810 14:58:17.556128 7429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:17.557641 7429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:58:17.561074 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 7336
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.253@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
I20260810 14:58:17.567185 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.558400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.28.193.254:44547)
0810 14:58:17.558510 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.558513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.558654 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:58:17.559078 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.559079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.559089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.559141 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.559695 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.560181 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.561251 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.561513 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.561584 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.563803 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.563805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.563805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.563813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.565231 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.565617 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.565619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.565620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.565699 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.566509 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.566510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.566511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.566657 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.566723 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.567017 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.567084 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:58:17.567384 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.558368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53842)
0810 14:58:17.558538 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.558573 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.559002 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.559201 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.559204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.559448 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.559633 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.559636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.560239 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.560239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.561129 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.561132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.561585 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.561586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.561756 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.562218 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.562227 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.563713 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.565315 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.565323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.565325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.565548 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.566308 (+ 760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.566309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.566309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.566442 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.566750 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.566758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.566886 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.566984 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.567146 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260810 14:58:17.567994 7429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:58:17.569589 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.561548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.28.193.253:44487)
0810 14:58:17.561743 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.561748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.561774 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:58:17.561808 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.561810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.561830 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.561893 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.562464 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.563392 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.564188 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.564476 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.564537 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.566691 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.566698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.566699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.566704 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.568301 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.568647 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.568650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.568650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.568709 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.568907 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.568907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.568908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.569074 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.569302 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.569346 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.569469 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.253@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
I20260810 14:58:17.574642 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.568237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39780)
0810 14:58:17.568438 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.568448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.568510 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.568649 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.568650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.568762 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.568834 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.568836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.569757 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.569758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.570366 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.570368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.570689 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.570690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.570735 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.571031 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.571035 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.572054 (+ 1019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.573922 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.573923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.573923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.574079 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.574229 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.574229 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.574230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.574276 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.574430 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.574430 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.574462 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.574517 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.574585 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260810 14:58:17.574752 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.568301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.28.193.253:44487)
0810 14:58:17.568441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.568443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.568475 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:58:17.568555 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.568556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.568565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.568626 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.568931 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.569604 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.570438 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.570641 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.570688 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.572122 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.572123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.572123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.572129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.573854 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.574142 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.574142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.574143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.574187 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.574310 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.574310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.574310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.574391 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.574431 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.574534 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.574630 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260810 14:58:17.575942 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Bootstrap starting.
I20260810 14:58:17.576566 7429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:17.576825 7429 log.cc:824] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:17.577512 7429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: No bootstrap required, opened a new log
I20260810 14:58:17.578949 7429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } }
I20260810 14:58:17.579073 7429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:17.579094 7429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2547cca3d9af436b9e5bc4164c20f7d1, State: Initialized, Role: FOLLOWER
I20260810 14:58:17.579214 7429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } }
I20260810 14:58:17.579519 7429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:17.579478 7442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } } }
I20260810 14:58:17.579576 7442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:58:17.581210 7463 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:17.581266 7463 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:17.582690 7336 master_runner.cc:428] Detected that this master 2547cca3d9af436b9e5bc4164c20f7d1 is joining an existing cluster
I20260810 14:58:17.582763 7336 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.193.253:44487
I20260810 14:58:17.590279 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.583749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39792)
0810 14:58:17.583866 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.583874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.583923 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.584057 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.584058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.584348 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.584438 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.584439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.585353 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.585354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.586033 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.586035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.586415 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.586416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.586479 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.586854 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.586858 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.587212 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.589041 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.589042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.589043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.589497 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.589669 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.589670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.589670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.589746 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.589944 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.589944 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.590002 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.590059 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.590164 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260810 14:58:17.590286 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.583707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.28.193.253:44487)
0810 14:58:17.583788 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.583789 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.583918 (+ 129us) server_negotiation.cc:407] Connection header received
0810 14:58:17.583986 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.583987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.583995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.584017 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.584517 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.585277 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.586108 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.586334 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.586398 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.587295 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.587296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.587296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.587303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.588974 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.589559 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.589560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.589560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.589614 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.589789 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.589790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.589790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.589892 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.589945 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.590087 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.590216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":63}
I20260810 14:58:17.591629 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.583714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.28.193.254:44547)
0810 14:58:17.583757 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.583759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.583883 (+ 124us) server_negotiation.cc:407] Connection header received
0810 14:58:17.583994 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.583995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.584008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.584070 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.585389 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.586183 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.587206 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.587480 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.587538 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.588674 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.588675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.588676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.588682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.590179 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.590982 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.590982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.590983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.591037 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.591219 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.591219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.591219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.591310 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.591353 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.591505 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.591567 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:58:17.591907 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.583663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53852)
0810 14:58:17.583821 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.583830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.583910 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.584078 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.584079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.584352 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.584434 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.584436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.586279 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.586280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.587129 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.587132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.587593 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.587594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.587658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.588282 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.588287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.588594 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.590281 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.590282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.590282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.590903 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.591100 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.591100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.591101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.591163 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.591359 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.591360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.591402 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.591488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.591587 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.254@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
I20260810 14:58:17.600946 7438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.594196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.28.193.253:44487)
0810 14:58:17.594244 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.594245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.594384 (+ 139us) server_negotiation.cc:407] Connection header received
0810 14:58:17.594474 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.594474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.594482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.594526 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.595012 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.595430 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.596500 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.596660 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.596701 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.598751 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.598752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.598753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.598759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.599954 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.600347 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.600348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.600348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.600388 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.600552 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.600552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.600552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.600633 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.600674 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.600838 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.600892 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:17.601068 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.594122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39806)
0810 14:58:17.594267 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.594318 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.594397 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.594544 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.594547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.594892 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.594949 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.594952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.595505 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.595506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.596431 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.596435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.596717 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.596718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.596814 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.597186 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.597191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.598673 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.600026 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.600033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.600035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.600284 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.600438 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.600439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.600439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.600500 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.600681 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.600687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.600737 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.600823 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.600954 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260810 14:58:17.601800 6601 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.193.253:44487: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
member_type: NON_VOTER
last_known_addr {
host: "127.28.193.253"
port: 44487
}
attrs {
promote: true
}
}
dest_uuid: "139086d48a2f416e870b433ec4c427cc"
cas_config_opid_index: -1
I20260810 14:58:17.602093 6601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } }
I20260810 14:58:17.602559 7478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } }
I20260810 14:58:17.602842 7394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:17.603622 7478 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.193.253:44487
I20260810 14:58:17.603771 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } }
I20260810 14:58:17.603844 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:17.603925 7336 master.cc:537] Master@127.28.193.253:44487 shutting down...
I20260810 14:58:17.604141 7394 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 14:58:17.604452 7478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:58:17.605224 6576 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 14:58:17.605624 6576 consensus_peers.cc:487] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc -> Peer 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253:44487): Couldn't send request to peer 2547cca3d9af436b9e5bc4164c20f7d1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:58:17.606665 7336 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:17.606751 7336 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:17.606802 7336 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: stopping tablet replica
I20260810 14:58:17.619211 7336 master.cc:552] Master@127.28.193.253:44487 shutdown complete.
I20260810 14:58:17.619266 7336 master_runner.cc:305] Clearing existing system tablet
I20260810 14:58:17.619395 7336 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.619437 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.620046 7336 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.620568 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.620700 7336 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.620815 7336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.620844 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "f2d6265a58b561b9f849bb8dbe07c784"
server_key_iv: "ad0f391e7b4f10cff90ff335ac16a7e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.621537 7336 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:17.623019 7336 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:58:17.623106 7336 log.cc:1196] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 14:58:17.623303 7336 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Deleting consensus metadata
I20260810 14:58:17.623417 7336 master_runner.cc:315] Copying system tablet from 127.28.193.254:44547
I20260810 14:58:17.623978 7336 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: tablet copy: Beginning tablet copy session from remote peer at address 127.28.193.254:44547
I20260810 14:58:17.631596 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.624255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53866)
0810 14:58:17.624559 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.624571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.624637 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.624792 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.624792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.625084 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.625146 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.625149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.626842 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.626843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.627434 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.627436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.628047 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.628048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.628104 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.628545 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.628549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.628850 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.630559 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.630560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.630561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.630739 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.630979 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.630980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.630980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.631045 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.631319 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.631320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.631361 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.631421 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.631516 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260810 14:58:17.631630 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.624510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.28.193.254:44547)
0810 14:58:17.624566 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.624568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.624640 (+ 72us) server_negotiation.cc:407] Connection header received
0810 14:58:17.624686 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.624687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.624698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.624774 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.625312 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.626734 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.627584 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.627951 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.628027 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.628926 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.628927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.628927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.628934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.630470 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.630831 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.630832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.630832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.630899 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.631108 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.631108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.631109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.631240 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.631315 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.631456 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.631533 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:17.632122 6634 tablet_copy_service.cc:140] P 139086d48a2f416e870b433ec4c427cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2547cca3d9af436b9e5bc4164c20f7d1 ({username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:53866)
I20260810 14:58:17.632222 6634 tablet_copy_service.cc:161] P 139086d48a2f416e870b433ec4c427cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2547cca3d9af436b9e5bc4164c20f7d1 at {username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:53866: session id = 2547cca3d9af436b9e5bc4164c20f7d1-00000000000000000000000000000000
I20260810 14:58:17.632860 6634 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:58:17.633416 7336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:58:17.634392 7336 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: tablet copy: Starting download of 0 data blocks...
I20260810 14:58:17.634521 7336 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: tablet copy: Starting download of 1 WAL segments...
I20260810 14:58:17.635385 7336 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:58:17.636168 6634 tablet_copy_service.cc:342] P 139086d48a2f416e870b433ec4c427cc: Request end of tablet copy session 2547cca3d9af436b9e5bc4164c20f7d1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:53866
I20260810 14:58:17.636220 6634 tablet_copy_service.cc:433] P 139086d48a2f416e870b433ec4c427cc: ending tablet copy session 2547cca3d9af436b9e5bc4164c20f7d1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2547cca3d9af436b9e5bc4164c20f7d1
I20260810 14:58:17.638588 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.639671 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.639681 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.639892 7336 server_base.cc:1034] running on GCE node
W20260810 14:58:17.641827 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.641990 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.642130 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.643265 7336 hybrid_clock.cc:648] HybridClock initialized: now 1786373897643252 us; error 29 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.644593 7336 init.cc:377] Logged in from keytab as oryx/127.28.193.253@KRBTEST.COM (short username oryx)
I20260810 14:58:17.644845 7336 webserver.cc:460] Webserver started at http://127.28.193.253:32825/ using document root <none> and password file <none>
I20260810 14:58:17.644964 7336 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.644999 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.645574 7336 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.646091 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.646219 7336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.646284 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "f2d6265a58b561b9f849bb8dbe07c784"
server_key_iv: "ad0f391e7b4f10cff90ff335ac16a7e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.646343 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.659153 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.660689 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.663872 7336 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.253:44487
I20260810 14:58:17.664075 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.253:44487 every 8 connection(s)
I20260810 14:58:17.664278 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
I20260810 14:58:17.665014 7597 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:58:17.665398 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Bootstrap starting.
I20260810 14:58:17.667645 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:58:17.667846 7597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Bootstrap complete.
I20260810 14:58:17.668020 7597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } }
I20260810 14:58:17.668087 7597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2547cca3d9af436b9e5bc4164c20f7d1, State: Initialized, Role: LEARNER
I20260810 14:58:17.668121 7597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } }
I20260810 14:58:17.668288 7597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:17.668291 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } }
I20260810 14:58:17.668426 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:58:17.669629 7617 catalog_manager.cc:1259] Loaded cluster ID: 217a14952eb94220ba367531acc804f5
I20260810 14:58:17.669673 7617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: loading cluster ID for follower catalog manager: success
I20260810 14:58:17.670598 7617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: acquiring CA information for follower catalog manager: success
I20260810 14:58:17.670861 7617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:58:17.677994 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.671138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.28.193.253:44487)
0810 14:58:17.671236 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.671238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.671313 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:58:17.671383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.671384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.671393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.671452 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.672100 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.672708 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.673761 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.673994 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.674053 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.675434 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.675435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.675435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.675442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.676774 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.677144 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.677145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.677145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.677190 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.677405 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.677407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.677407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.677525 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.677618 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.677800 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.677873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.28.193.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.252.keytab.
Entry for principal oryx/127.28.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.252.keytab.
I20260810 14:58:17.706588 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:44605
--webserver_interface=127.28.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.252.keytab
--principal=oryx/127.28.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:17.799357 7624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:17.799556 7624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:17.799575 7624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:17.800709 7624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:17.800751 7624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:17.800763 7624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:17.800774 7624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:17.800786 7624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:17.802340 7624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.252.keytab
--principal=oryx/127.28.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:44605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:17.802564 7624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:17.802788 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.807492 7630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.807626 7624 server_base.cc:1034] running on GCE node
W20260810 14:58:17.807487 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.807523 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.807960 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.808287 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.809453 7624 hybrid_clock.cc:648] HybridClock initialized: now 1786373897809434 us; error 30 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.812433 7624 init.cc:377] Logged in from keytab as oryx/127.28.193.252@KRBTEST.COM (short username oryx)
I20260810 14:58:17.812858 7624 webserver.cc:460] Webserver started at http://127.28.193.252:42853/ using document root <none> and password file <none>
I20260810 14:58:17.813071 7624 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.813117 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.813215 7624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:17.814358 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/instance:
uuid: "b8741d68b81a4a08ac8db88bd79de424"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "7a5d73fac266fc7ca8d8ba56ab7e1fd0"
server_key_iv: "c71fb9b01a06c5a0a43e970789ff89dc"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.814713 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal/instance:
uuid: "b8741d68b81a4a08ac8db88bd79de424"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "7a5d73fac266fc7ca8d8ba56ab7e1fd0"
server_key_iv: "c71fb9b01a06c5a0a43e970789ff89dc"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.816087 7624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:17.816776 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.816932 7624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.817008 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "b8741d68b81a4a08ac8db88bd79de424"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "7a5d73fac266fc7ca8d8ba56ab7e1fd0"
server_key_iv: "c71fb9b01a06c5a0a43e970789ff89dc"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.817075 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.837332 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.839146 7624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:17.839267 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.842993 7624 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.252:44605
I20260810 14:58:17.843034 7721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.252:44605 every 8 connection(s)
I20260810 14:58:17.843345 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
I20260810 14:58:17.844316 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:17.846045 7723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.252@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:17.852691 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 7624
I20260810 14:58:17.854353 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.846867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.28.193.254:44547)
0810 14:58:17.846921 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.846923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.846982 (+ 59us) server_negotiation.cc:407] Connection header received
0810 14:58:17.847474 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.847475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.847485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.847542 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.848115 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.848607 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.849495 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.849669 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.849728 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.851689 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.851690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.851690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.851696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.853022 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.853480 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.853481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.853482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.853549 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.853777 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.853778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.853778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.853919 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.853987 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.854224 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.854289 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:58:17.854581 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.846697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53870)
0810 14:58:17.846862 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.846902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.847383 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.847602 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.847606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.847863 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.848051 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.848055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.848664 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.848665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.849434 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.849435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.849785 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.849786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.849883 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.850321 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.850329 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.851611 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.853095 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.853103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.853105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.853378 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.853606 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.853607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.853607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.853702 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.853981 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.853985 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.854093 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.854198 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.854353 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":101,"threads_started":1}
I20260810 14:58:17.855125 7723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.252@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
I20260810 14:58:17.861083 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.853159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.28.193.252:44605)
0810 14:58:17.853324 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.853329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.853347 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:58:17.853368 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.853370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.853383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.853448 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.854139 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.855084 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.856014 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.856295 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.856351 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.858589 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.858596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.858596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.858601 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.859989 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860315 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.860319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.860319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.860378 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860551 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.860552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.860552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.860712 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.860862 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.860907 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.860983 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260810 14:58:17.861159 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.855423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.28.193.253:44487)
0810 14:58:17.855479 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.855481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.855489 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:17.855520 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.855521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.855531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.855574 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.855830 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.856295 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.857074 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.857286 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.857340 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.858968 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.858969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.858970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.858977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.860200 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860486 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.860487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.860487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.860533 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860692 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.860694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.860694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.860811 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.860871 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.861006 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.861080 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:58:17.861274 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.855354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39830)
0810 14:58:17.855400 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.855410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.855466 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.855578 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.855579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.855720 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.855774 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.855775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.856383 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.856385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.856996 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.856998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.857343 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.857344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.857410 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.857761 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.857766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.858903 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.860266 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.860267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.860418 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.860593 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.860594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.860594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.860642 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.860871 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.860871 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.860910 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.860976 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.861058 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 14:58:17.861676 7723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.252@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
I20260810 14:58:17.867121 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.861911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43380)
0810 14:58:17.861954 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.861965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.862009 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.862118 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.862119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.862237 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.862261 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.862262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.862819 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.862819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.863265 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.863266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.863588 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.863589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.863633 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.863915 (+ 282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.863918 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.864930 (+ 1012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.866330 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.866330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.866331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.866514 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.866694 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.866695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.866695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.866741 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.866903 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.866904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.866934 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.866975 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.867044 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 14:58:17.867182 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.861937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.28.193.252:44605)
0810 14:58:17.861968 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.861970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.861996 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:58:17.862057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.862057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.862066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.862115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.862318 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.862755 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.863329 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.863533 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.863584 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.865004 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.865005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.865006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.865012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.866256 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.866598 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.866599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.866600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.866650 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.866780 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.866781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.866781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.866863 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.866905 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.866998 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.867088 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":18}
I20260810 14:58:17.868388 7723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Bootstrap starting.
I20260810 14:58:17.869251 7723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:17.869513 7723 log.cc:824] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:17.870239 7723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: No bootstrap required, opened a new log
I20260810 14:58:17.871652 7723 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } }
I20260810 14:58:17.871794 7723 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:17.871831 7723 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8741d68b81a4a08ac8db88bd79de424, State: Initialized, Role: FOLLOWER
I20260810 14:58:17.871948 7723 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } }
I20260810 14:58:17.872227 7723 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:17.872179 7736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } } }
I20260810 14:58:17.872270 7736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:58:17.873868 7756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:17.873921 7756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:17.875887 7624 master_runner.cc:428] Detected that this master b8741d68b81a4a08ac8db88bd79de424 is joining an existing cluster
I20260810 14:58:17.875946 7624 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.193.252:44605
I20260810 14:58:17.883944 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43384)
0810 14:58:17.877966 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.877974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.878036 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.878164 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.878164 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.878357 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.878400 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.878401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879302 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.879302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880181 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.880183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880652 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880705 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.881148 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.881152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.881454 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.883054 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.883055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.883055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.883210 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.883389 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.883390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.883390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.883432 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.883640 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.883641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.883696 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.883754 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.883841 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260810 14:58:17.884078 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.28.193.252:44605)
0810 14:58:17.877832 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.877833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.878011 (+ 178us) server_negotiation.cc:407] Connection header received
0810 14:58:17.878088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.878088 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.878096 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.878119 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.878481 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879204 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880266 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880543 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880578 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.881538 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.881539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.881539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.881546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.882979 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.883273 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.883274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.883274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.883337 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.883469 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.883470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.883470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.883589 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.883652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.883820 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.883924 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 14:58:17.902031 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39834)
0810 14:58:17.877784 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.877792 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.877879 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.878038 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.878038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.878296 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.878363 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.878365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879295 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.879295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880181 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.880183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880673 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880736 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.898251 (+ 17515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.898257 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.898648 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.900845 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.900846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.900846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.901093 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.901287 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.901288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.901347 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.901656 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.901657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.901745 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.901814 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.901924 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260810 14:58:17.902420 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.28.193.254:44547)
0810 14:58:17.877642 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.877644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.877652 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:17.877770 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.877771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.877782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.877840 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.878318 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879120 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880296 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880569 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880612 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.899278 (+ 18666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.899280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.899280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.899289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.900768 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901291 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.901292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.901292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.901345 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901863 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.901864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.901865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.902009 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.902053 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.902244 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.902326 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:58:17.902559 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.28.193.253:44487)
0810 14:58:17.877612 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.877614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.877836 (+ 222us) server_negotiation.cc:407] Connection header received
0810 14:58:17.877926 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.877927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.877939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.877986 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.878483 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879205 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880274 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880574 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880608 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.898764 (+ 18156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.898766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.898766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.898776 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.900260 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901161 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.901162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.901163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.901219 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901413 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.901415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.901415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.901593 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.901660 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.902367 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.902466 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:58:17.902830 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.877426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53886)
0810 14:58:17.877606 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.877617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.877663 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.877888 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.877889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.878151 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.878223 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.878225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.879242 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.879242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880196 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.880199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.880690 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.880691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.880753 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.898727 (+ 17974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.898732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.899126 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.900848 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.900850 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.900850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.901204 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.901692 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.901693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.901693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.901784 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.902099 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.902100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.902142 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.902564 (+ 422us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.902723 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":69,"thread_start_us":87,"threads_started":1}
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373896, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.254@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
I20260810 14:58:17.912300 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.905999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.28.193.252:44605)
0810 14:58:17.906035 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.906036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.906064 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:58:17.906095 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.906096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.906104 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.906139 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.906421 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.906864 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.907832 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.908016 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.908070 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.909849 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.909851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.909851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.909857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.911150 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.911524 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.911525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.911525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.911576 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.911772 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.911773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.911773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.911866 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.911930 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.912133 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.912211 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 14:58:17.912413 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 6700
I20260810 14:58:17.912563 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.905925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43396)
0810 14:58:17.905973 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.905985 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.906026 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.906154 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.906155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.906316 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.906362 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.906364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.906939 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.906940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.907763 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.907765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.908073 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.908074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.908136 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.908445 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.908449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.909775 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.911237 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.911238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.911238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.911441 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.911661 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.911662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.911662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.911723 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.911916 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.911916 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.911969 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.912030 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.912470 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 14:58:17.932545 6601 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.193.252:44605: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424"
member_type: NON_VOTER
last_known_addr {
host: "127.28.193.252"
port: 44605
}
attrs {
promote: true
}
}
dest_uuid: "139086d48a2f416e870b433ec4c427cc"
cas_config_opid_index: 5
I20260810 14:58:17.932852 6601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } }
I20260810 14:58:17.933415 7478 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.933502 7688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:17.934381 7478 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.193.252:44605
I20260810 14:58:17.934427 7688 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:58:17.934455 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.934518 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:17.934747 7624 master.cc:537] Master@127.28.193.252:44605 shutting down...
I20260810 14:58:17.934945 7479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:58:17.935950 6576 consensus_peers.cc:487] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc -> Peer b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605): Couldn't send request to peer b8741d68b81a4a08ac8db88bd79de424. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 14:58:17.937780 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:45807
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:44547
--webserver_port=33893
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:58:17.938704 7624 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:17.938794 7624 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:17.938835 7624 tablet_replica.cc:330] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: stopping tablet replica
I20260810 14:58:17.939862 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.933372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39846)
0810 14:58:17.936340 (+ 2968us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.936352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.936403 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.936618 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.936619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.936621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:17.936668 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.936670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.937576 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.937577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.938761 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.938764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.939610 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.939611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.939672 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.939698 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.939753 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.939795 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2714}
I20260810 14:58:17.939885 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.934744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.28.193.253:44487)
0810 14:58:17.936319 (+ 1575us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.936321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.936451 (+ 130us) server_negotiation.cc:407] Connection header received
0810 14:58:17.936478 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.936479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.936505 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.936557 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:17.936735 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.937480 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.938856 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.939421 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.939492 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.939572 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.939767 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.939818 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546}
I20260810 14:58:17.940364 7557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 LEARNER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:58:17.940834 7479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 14:58:17.941411 7557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.942658 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139086d48a2f416e870b433ec4c427cc. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.942762 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:58:17.942802 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: NON_VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: true } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.942852 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:58:17.942857 7479 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: attempting to promote NON_VOTER 2547cca3d9af436b9e5bc4164c20f7d1 to VOTER
I20260810 14:58:17.943068 7479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } }
I20260810 14:58:17.943636 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.943497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43398)
0810 14:58:17.943554 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.943606 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:44605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:58:17.943662 7557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 LEARNER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 14:58:17.943853 7478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 14:58:17.944098 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.943974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43402)
0810 14:58:17.944049 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.944072 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:44605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 14:58:17.944168 6576 consensus_peers.cc:487] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc -> Peer b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605): Couldn't send request to peer b8741d68b81a4a08ac8db88bd79de424. Status: Network error: Client connection negotiation failed: client connection to 127.28.193.252:44605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:58:17.944489 7479 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.945411 7479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.945487 7479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:17.945451 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.945535 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:17.945827 7557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.946666 7600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:17.946744 7600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:58:17.952574 7624 master.cc:552] Master@127.28.193.252:44605 shutdown complete.
I20260810 14:58:17.952620 7624 master_runner.cc:305] Clearing existing system tablet
I20260810 14:58:17.952749 7624 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.952800 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.953548 7624 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:17.954103 7782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.954255 7624 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.954308 7624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.954329 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "b8741d68b81a4a08ac8db88bd79de424"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "7a5d73fac266fc7ca8d8ba56ab7e1fd0"
server_key_iv: "c71fb9b01a06c5a0a43e970789ff89dc"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.955150 7624 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:17.956454 7624 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:58:17.956526 7624 log.cc:1196] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 14:58:17.956708 7624 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Deleting consensus metadata
I20260810 14:58:17.956856 7624 master_runner.cc:315] Copying system tablet from 127.28.193.254:44547
I20260810 14:58:17.957453 7624 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: tablet copy: Beginning tablet copy session from remote peer at address 127.28.193.254:44547
I20260810 14:58:17.964898 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.957970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.28.193.254:44547)
0810 14:58:17.958019 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:17.958022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:17.958075 (+ 53us) server_negotiation.cc:407] Connection header received
0810 14:58:17.958157 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:17.958158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:17.958169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:17.958233 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:17.958612 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.959386 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.960440 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.960734 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.960811 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.962053 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:17.962055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:17.962055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:17.962063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:17.963689 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.964100 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.964101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.964101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.964162 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:17.964365 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:17.964366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:17.964366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:17.964491 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:17.964554 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:17.964729 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 14:58:17.964809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 14:58:17.964905 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:17.957869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:53894)
0810 14:58:17.958013 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:17.958024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:17.958091 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:17.958240 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:17.958241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:17.958454 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:17.958532 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.958535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.959478 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.959480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.960324 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:17.960327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:17.960826 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:17.960827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:17.960899 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:17.961488 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:17.961493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:17.961947 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:17.963779 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.963781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.963781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.964015 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.964225 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:17.964225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:17.964226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:17.964296 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:17.964560 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:17.964561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:17.964620 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 14:58:17.964697 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:58:17.964805 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260810 14:58:17.965551 6634 tablet_copy_service.cc:140] P 139086d48a2f416e870b433ec4c427cc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b8741d68b81a4a08ac8db88bd79de424 ({username='oryx', principal='oryx/127.28.193.252@KRBTEST.COM'} at 127.0.0.1:53894)
I20260810 14:58:17.965619 6634 tablet_copy_service.cc:161] P 139086d48a2f416e870b433ec4c427cc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b8741d68b81a4a08ac8db88bd79de424 at {username='oryx', principal='oryx/127.28.193.252@KRBTEST.COM'} at 127.0.0.1:53894: session id = b8741d68b81a4a08ac8db88bd79de424-00000000000000000000000000000000
I20260810 14:58:17.966324 6634 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:58:17.966852 7624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:58:17.968166 7624 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: tablet copy: Starting download of 0 data blocks...
I20260810 14:58:17.968308 7624 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: tablet copy: Starting download of 1 WAL segments...
I20260810 14:58:17.969030 7624 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:58:17.970014 6634 tablet_copy_service.cc:342] P 139086d48a2f416e870b433ec4c427cc: Request end of tablet copy session b8741d68b81a4a08ac8db88bd79de424-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.193.252@KRBTEST.COM'} at 127.0.0.1:53894
I20260810 14:58:17.970067 6634 tablet_copy_service.cc:433] P 139086d48a2f416e870b433ec4c427cc: ending tablet copy session b8741d68b81a4a08ac8db88bd79de424-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b8741d68b81a4a08ac8db88bd79de424
I20260810 14:58:17.972553 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:17.973788 7792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.973820 7795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:17.978529 7793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:17.978729 7624 server_base.cc:1034] running on GCE node
I20260810 14:58:17.978828 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:17.978963 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:17.980075 7624 hybrid_clock.cc:648] HybridClock initialized: now 1786373897980067 us; error 23 us; skew 500 ppm
Aug 10 14:58:17 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373897, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:17.981478 7624 init.cc:377] Logged in from keytab as oryx/127.28.193.252@KRBTEST.COM (short username oryx)
I20260810 14:58:17.981765 7624 webserver.cc:460] Webserver started at http://127.28.193.252:33993/ using document root <none> and password file <none>
I20260810 14:58:17.981890 7624 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:17.981926 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:17.982630 7624 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:17.983189 7811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:17.983340 7624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:17.983386 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "b8741d68b81a4a08ac8db88bd79de424"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "7a5d73fac266fc7ca8d8ba56ab7e1fd0"
server_key_iv: "c71fb9b01a06c5a0a43e970789ff89dc"
server_key_version: "encryptionkey@0"
I20260810 14:58:17.983440 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:17.991890 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:17.993626 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:17.996809 7624 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.252:44605
I20260810 14:58:17.996848 7890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.252:44605 every 8 connection(s)
I20260810 14:58:17.997274 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
I20260810 14:58:17.998210 7891 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:58:17.998726 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Bootstrap starting.
I20260810 14:58:18.001559 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:58:18.001817 7891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: Bootstrap complete.
I20260810 14:58:18.002012 7891 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } }
I20260810 14:58:18.002101 7891 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8741d68b81a4a08ac8db88bd79de424, State: Initialized, Role: LEARNER
I20260810 14:58:18.002135 7891 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } }
I20260810 14:58:18.002285 7891 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:18.003592 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:18.003696 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:58:18.004805 7911 catalog_manager.cc:1259] Loaded cluster ID: 217a14952eb94220ba367531acc804f5
I20260810 14:58:18.004858 7911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: loading cluster ID for follower catalog manager: success
I20260810 14:58:18.005968 7911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: acquiring CA information for follower catalog manager: success
I20260810 14:58:18.006259 7911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:58:18.033452 7773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:18.033654 7773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:18.033684 7773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:18.034832 7773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:18.034881 7773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:18.034914 7773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:18.036361 7773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:45807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=33893
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:18.036584 7773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:18.036796 7773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:18.041500 7919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:18.041512 7920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:18.041630 7773 server_base.cc:1034] running on GCE node
W20260810 14:58:18.041512 7923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:18.041940 7773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:18.042158 7773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:18.043315 7773 hybrid_clock.cc:648] HybridClock initialized: now 1786373898043295 us; error 38 us; skew 500 ppm
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:18.045787 7773 init.cc:377] Logged in from keytab as oryx/127.28.193.193@KRBTEST.COM (short username oryx)
I20260810 14:58:18.046151 7773 webserver.cc:460] Webserver started at http://127.28.193.193:33893/ using document root <none> and password file <none>
I20260810 14:58:18.046327 7773 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:18.046381 7773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:18.047613 7773 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:18.048110 7935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.048239 7773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.048321 7773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "d1cb14a973b591a2a31566324e6a49f7"
server_key_iv: "9633dd32c0eddb8d6d96f3817ab9317b"
server_key_version: "encryptionkey@0"
I20260810 14:58:18.048632 7773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:18.057984 7773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:18.059952 7773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:18.060072 7773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:18.060269 7773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:18.060541 7773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:18.060573 7773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.060595 7773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:18.060618 7773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.066164 7773 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45807
I20260810 14:58:18.066241 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45807 every 8 connection(s)
I20260810 14:58:18.066545 7773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:18.074400 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 7773
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
I20260810 14:58:18.074571 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 6907
I20260810 14:58:18.081038 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:48419 (local address 127.28.193.254:44547)
0810 14:58:18.067289 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.067291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.067524 (+ 233us) server_negotiation.cc:407] Connection header received
0810 14:58:18.068186 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.068187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.068199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.068242 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.069781 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070378 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071143 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.071435 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071500 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.074401 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.074402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.074403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.074410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.079474 (+ 5064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.079951 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.079952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.079953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.080015 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080292 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.080292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.080293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.080425 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.080484 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.080863 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.080946 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 14:58:18.081491 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.28.193.193:55489)
0810 14:58:18.067629 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.067639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.068047 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.068610 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.068611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.068868 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.069147 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.069152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070300 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.070301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.071064 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.071066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.071423 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.071551 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.072161 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.072164 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.074747 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.080265 (+ 5518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.080266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.080449 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.080643 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.080644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.080697 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.080984 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.080985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.081025 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.081094 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.081268 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":104,"thread_start_us":61,"threads_started":1}
I20260810 14:58:18.081840 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:55489 (local address 127.28.193.252:44605)
0810 14:58:18.067736 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.067738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.067748 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:18.068132 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.068133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.068147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.068210 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.069231 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070234 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071142 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.071357 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071413 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.074840 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.074841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.074841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.074848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.080188 (+ 5340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080524 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.080525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.080526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.080582 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080758 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.080759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.080759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.080884 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.080957 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.081640 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.081747 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":50,"threads_started":1}
I20260810 14:58:18.081845 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:48419)
0810 14:58:18.067385 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.067434 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.068045 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.068306 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.068306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.068864 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.069147 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.069154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070462 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.070462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.071066 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.071067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.071497 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.071552 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.072102 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.072112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.074301 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.079554 (+ 5253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.079563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.079566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.079854 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.080088 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.080088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.080089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.080218 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.080530 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.080537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.080699 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.081384 (+ 685us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.081486 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":109,"thread_start_us":76,"threads_started":1}
I20260810 14:58:18.082520 8121 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:18.082617 8121 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.082803 8121 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:18.083238 6601 ts_manager.cc:194] Re-registered known tserver with Master: fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807)
I20260810 14:58:18.083606 6601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:48419
I20260810 14:58:18.087324 8120 heartbeater.cc:347] Connected to a master server at 127.28.193.252:44605
I20260810 14:58:18.087395 8120 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.087472 8120 heartbeater.cc:511] Master 127.28.193.252:44605 requested a full tablet report, sending...
I20260810 14:58:18.088373 7837 ts_manager.cc:194] Registered new tserver with Master: fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807)
I20260810 14:58:18.089196 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:37059 (local address 127.28.193.253:44487)
0810 14:58:18.067289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.067291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.067524 (+ 233us) server_negotiation.cc:407] Connection header received
0810 14:58:18.068164 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.068166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.068179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.068210 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.069273 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070025 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.071795 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.071983 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.072039 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.073929 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.073931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.073931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.073938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.088112 (+ 14174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.088500 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.088501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.088502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.088569 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.088748 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.088749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.088749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.088887 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.088935 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.089076 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.089136 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:18.089268 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.067212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.28.193.193:37059)
0810 14:58:18.067385 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.067434 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.068039 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.068300 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.068305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.068844 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.069147 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.069154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.070150 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.070151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.071731 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.071733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.072059 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.072061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.072143 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.072470 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.072475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.073832 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.088195 (+ 14363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.088196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.088197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.088408 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.088630 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.088631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.088631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.088695 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.088938 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.088939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.088989 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.089058 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.089165 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":749,"thread_start_us":86,"threads_started":1}
I20260810 14:58:18.089612 8122 heartbeater.cc:347] Connected to a master server at 127.28.193.253:44487
I20260810 14:58:18.089655 8122 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.089689 8122 heartbeater.cc:511] Master 127.28.193.253:44487 requested a full tablet report, sending...
I20260810 14:58:18.090094 7542 ts_manager.cc:194] Registered new tserver with Master: fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807)
I20260810 14:58:18.091921 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:46199
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:44547
--webserver_port=41357
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:18.174638 8134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:18.174865 8134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:18.174890 8134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:18.176007 8134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:18.176049 8134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:18.176075 8134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:18.177543 8134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:46199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=41357
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:18.177778 8134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:18.177985 8134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:18.183101 8134 server_base.cc:1034] running on GCE node
W20260810 14:58:18.183084 8145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:18.183128 8146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:18.183086 8148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:18.183554 8134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:18.183811 8134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:18.184981 8134 hybrid_clock.cc:648] HybridClock initialized: now 1786373898184968 us; error 29 us; skew 500 ppm
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:18.187536 8134 init.cc:377] Logged in from keytab as oryx/127.28.193.194@KRBTEST.COM (short username oryx)
I20260810 14:58:18.187959 8134 webserver.cc:460] Webserver started at http://127.28.193.194:41357/ using document root <none> and password file <none>
I20260810 14:58:18.188154 8134 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:18.188215 8134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:18.189502 8134 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:18.190093 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.190237 8134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.190316 8134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "605ef25bc3ea4c84855d440bbfd6903d"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "2c16ed735c4b8ed065d2893a7447067f"
server_key_iv: "0e8e559b5df9b474180eab1f12d2d99d"
server_key_version: "encryptionkey@0"
I20260810 14:58:18.190658 8134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:18.212129 8134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:18.214717 8134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:18.214860 8134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:18.215109 8134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:18.215478 8134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:18.215528 8134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.215559 8134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:18.215576 8134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.225641 8134 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:46199
I20260810 14:58:18.225683 8340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:46199 every 8 connection(s)
I20260810 14:58:18.226092 8134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:18.227442 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 8134
I20260810 14:58:18.227588 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 7117
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
I20260810 14:58:18.241022 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.226866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:44729 (local address 127.28.193.254:44547)
0810 14:58:18.226915 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.226917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.227209 (+ 292us) server_negotiation.cc:407] Connection header received
0810 14:58:18.230129 (+ 2920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.230131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.230144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.230170 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.231356 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.231971 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.233185 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.233411 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.233445 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.236194 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.236196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.236197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.236205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.238769 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.239721 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.239723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.239723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.239787 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.240247 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.240248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.240249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.240409 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.240477 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.240841 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.240924 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 14:58:18.241215 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.226890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:44729)
0810 14:58:18.227069 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.227114 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.230032 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.230669 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.230673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.231004 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.231259 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.231265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.232065 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.232066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.233097 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.233099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.233516 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.233517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.233634 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.234232 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.234244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.236082 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.239322 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.239331 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.239334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.239630 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.239858 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.239859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.239859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.240014 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.240507 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.240515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.240683 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.240797 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.240972 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1,"spinlock_wait_cycles":8832,"thread_start_us":93,"threads_started":1}
I20260810 14:58:18.241928 8343 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:18.242048 8343 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.242264 8343 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:18.242662 6601 ts_manager.cc:194] Re-registered known tserver with Master: 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
I20260810 14:58:18.242810 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.226929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:52425 (local address 127.28.193.253:44487)
0810 14:58:18.226976 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.226977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.227209 (+ 232us) server_negotiation.cc:407] Connection header received
0810 14:58:18.234303 (+ 7094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.234305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.234321 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.234353 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.234769 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.235479 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.236460 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.236664 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.236697 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.239335 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.239337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.239337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.239344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.240734 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.241838 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.241839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.241840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.241902 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.242213 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.242214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.242215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.242342 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.242395 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.242653 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.242727 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 14:58:18.243134 6601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:44729
I20260810 14:58:18.243348 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.226890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.28.193.194:52425)
0810 14:58:18.227066 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.227114 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.227679 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.234418 (+ 6739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.234419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.234602 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.234671 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.234673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.235598 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.235599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.236356 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.236359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.236768 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.236769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.236817 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.237209 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.237215 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.239246 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.241529 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.241530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.241531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.241755 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.241975 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.241975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.241976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.242136 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.242497 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.242498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.242539 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.243090 (+ 551us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.243236 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
I20260810 14:58:18.244524 8346 heartbeater.cc:347] Connected to a master server at 127.28.193.253:44487
I20260810 14:58:18.244577 8346 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.244643 8346 heartbeater.cc:511] Master 127.28.193.253:44487 requested a full tablet report, sending...
I20260810 14:58:18.244673 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:42045
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:44547
--webserver_port=34875
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:43507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:58:18.245996 7542 ts_manager.cc:194] Registered new tserver with Master: 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
I20260810 14:58:18.247682 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.238288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:48009 (local address 127.28.193.252:44605)
0810 14:58:18.238871 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.238872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.238881 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:18.238906 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.238907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.238918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.238972 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.240835 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.241589 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.242532 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.242815 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.242868 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.245047 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.245048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.245049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.245057 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.246641 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.246994 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.246994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.246995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.247047 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.247246 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.247247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.247247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.247352 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.247400 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.247554 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.247622 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552}
I20260810 14:58:18.247721 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.227181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.28.193.194:48009)
0810 14:58:18.227284 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.227294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.227754 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.239664 (+ 11910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.239666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.240181 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.240258 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.240261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.241673 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.241674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.242437 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.242441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.242920 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.242921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.242980 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.243345 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.243350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.244972 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.246711 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.246712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.246713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.246921 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.247102 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.247103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.247103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.247165 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.247414 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.247415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.247464 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.247528 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.247631 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":54,"threads_started":1}
I20260810 14:58:18.248021 8341 heartbeater.cc:347] Connected to a master server at 127.28.193.252:44605
I20260810 14:58:18.248075 8341 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.248122 8341 heartbeater.cc:511] Master 127.28.193.252:44605 requested a full tablet report, sending...
I20260810 14:58:18.248399 7837 ts_manager.cc:194] Registered new tserver with Master: 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
W20260810 14:58:18.327817 8351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:18.328022 8351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:18.328053 8351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:18.329166 8351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:18.329206 8351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:18.329239 8351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:18.330716 8351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:43507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:42045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=34875
--tserver_master_addrs=127.28.193.254:44547,127.28.193.253:44487,127.28.193.252:44605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:18.330957 8351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:18.331169 8351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:18.336405 8360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:18.336486 8361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:18.336558 8351 server_base.cc:1034] running on GCE node
W20260810 14:58:18.336405 8364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:18.336869 8351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:18.337118 8351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:18.338286 8351 hybrid_clock.cc:648] HybridClock initialized: now 1786373898338264 us; error 33 us; skew 500 ppm
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:18.340831 8351 init.cc:377] Logged in from keytab as oryx/127.28.193.195@KRBTEST.COM (short username oryx)
I20260810 14:58:18.341274 8351 webserver.cc:460] Webserver started at http://127.28.193.195:34875/ using document root <none> and password file <none>
I20260810 14:58:18.341480 8351 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:18.341547 8351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:18.342902 8351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:18.343442 8376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.343590 8351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:18.343672 8351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "6b1442114b594142b9112d03e7bdbc7c"
format_stamp: "Formatted at 2026-08-10 14:58:17 on dist-test-slave-cvwc"
server_key: "459b9fab6b3fe32749b08cfa479d166b"
server_key_iv: "a7a2383f3436f0677a69b8e4088a2cf8"
server_key_version: "encryptionkey@0"
I20260810 14:58:18.344026 8351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:18.365294 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.362320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43406)
0810 14:58:18.362389 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.362402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.362467 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.362624 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.362625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.362627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.362675 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.362677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.363568 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.363570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.364327 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.364330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.365069 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.365070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.365134 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.365137 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.365185 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.365227 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:58:18.365309 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.362392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.28.193.252:44605)
0810 14:58:18.362462 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.362465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.362477 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:58:18.362513 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.362514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.362541 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.362606 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.362723 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.363463 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.364406 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.364984 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.365050 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.365127 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.365199 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.365229 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:58:18.365666 7850 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 14:58:18.365875 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139086d48a2f416e870b433ec4c427cc. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:18.366000 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:58:18.372510 8351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:18.374361 8351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:18.374490 8351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:18.374722 8351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:18.375036 8351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:18.375069 8351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.375097 8351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:18.375120 8351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:18.380617 8351 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:42045
I20260810 14:58:18.380685 8553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:42045 every 8 connection(s)
I20260810 14:58:18.381018 8351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.253@KRBTEST.COM
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
Aug 10 14:58:18 dist-test-slave-cvwc krb5kdc[6522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373898, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.252@KRBTEST.COM
I20260810 14:58:18.390926 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 8351
I20260810 14:58:18.391362 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.381771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40121 (local address 127.28.193.253:44487)
0810 14:58:18.381821 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.381823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.382008 (+ 185us) server_negotiation.cc:407] Connection header received
0810 14:58:18.382605 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.382607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.382618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.382648 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.383479 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.383985 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385283 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385535 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385581 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.387770 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.387772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.387772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.387781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.389478 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390462 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.390519 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390699 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.390828 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.390884 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.391175 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.391265 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:58:18.391363 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.381684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44269 (local address 127.28.193.254:44547)
0810 14:58:18.381757 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.381760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.382028 (+ 268us) server_negotiation.cc:407] Connection header received
0810 14:58:18.382588 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.382589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.382601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.382650 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.383418 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.384220 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385107 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385336 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385399 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.388236 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.388238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.388238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.388246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.389836 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390289 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.390350 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390617 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.390760 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.390821 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.391195 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.391267 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:58:18.391464 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.382199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:50453 (local address 127.28.193.252:44605)
0810 14:58:18.382247 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.382249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.382256 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:18.382604 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.382605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.382618 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.382647 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.383455 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.384174 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385626 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385890 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385948 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.388694 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.388696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.388696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.388704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.390101 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390652 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.390706 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390916 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.390917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.390918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.391047 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.391089 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.391301 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.391382 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:58:18.391939 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.381696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.28.193.195:40121)
0810 14:58:18.381893 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.381934 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.382503 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.382738 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.382738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.383114 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.383336 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.383341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.384088 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.384089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.385192 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.385195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385599 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.385657 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.386212 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.386217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.387642 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.390194 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.390195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390380 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.390587 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.390588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390639 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.390907 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.390908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.391020 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.391523 (+ 503us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.391719 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260810 14:58:18.392168 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.382032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.28.193.195:50453)
0810 14:58:18.382157 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.382165 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.382504 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.383114 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.383114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.383247 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.383335 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.383341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.384245 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.384246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.385198 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.385200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385969 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.386027 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.386348 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.386353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.388620 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.390193 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.390194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390559 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.390791 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.390792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390846 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.391137 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.391137 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.391180 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.391573 (+ 393us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.392080 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":406,"thread_start_us":58,"threads_started":1}
I20260810 14:58:18.392421 8559 heartbeater.cc:347] Connected to a master server at 127.28.193.253:44487
I20260810 14:58:18.392527 8559 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.392735 8559 heartbeater.cc:511] Master 127.28.193.253:44487 requested a full tablet report, sending...
I20260810 14:58:18.393064 8554 heartbeater.cc:347] Connected to a master server at 127.28.193.252:44605
I20260810 14:58:18.393123 8554 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.393196 8554 heartbeater.cc:511] Master 127.28.193.252:44605 requested a full tablet report, sending...
I20260810 14:58:18.393231 7542 ts_manager.cc:194] Registered new tserver with Master: 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:18.393410 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.381696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:44269)
0810 14:58:18.381893 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.381933 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.382493 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.382719 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.382724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.383113 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:18.383335 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.383341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.384316 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.384317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.385029 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.385031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.385408 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.385408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.385522 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.386089 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:18.386098 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:18.388132 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:18.389913 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.389920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.389923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390214 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.390413 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:18.390413 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:18.390413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:18.390543 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:18.390849 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:18.390856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:18.391056 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.392174 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.393044 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"spinlock_wait_cycles":384,"thread_start_us":112,"threads_started":1}
I20260810 14:58:18.393538 7837 ts_manager.cc:194] Registered new tserver with Master: 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:18.393780 8555 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44547
I20260810 14:58:18.393831 8555 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:18.393900 8555 heartbeater.cc:511] Master 127.28.193.254:44547 requested a full tablet report, sending...
I20260810 14:58:18.394120 6601 ts_manager.cc:194] Re-registered known tserver with Master: 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:18.394776 6601 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:44269
I20260810 14:58:18.400722 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.392910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.28.193.252:44605)
0810 14:58:18.392949 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.392951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.393887 (+ 936us) server_negotiation.cc:407] Connection header received
0810 14:58:18.393936 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.393937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.393947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.393997 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.394627 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.395354 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396463 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.396741 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396788 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.398380 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.398381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.398381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.398388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.399600 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.399932 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.399933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.399933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.399984 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.400175 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.400176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.400177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.400321 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.400390 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.400575 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.400639 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:58:18.400924 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.392737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.28.193.253:44487)
0810 14:58:18.392783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.392785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.393792 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 14:58:18.394032 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.394033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.394043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.394069 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.394736 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.395396 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396178 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.396371 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396425 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.397754 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.397755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.397756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.397762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.399624 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.400046 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.400047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.400047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.400169 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.400370 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.400370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.400371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.400487 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.400556 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.400768 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.400843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:58:18.401165 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.392426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.28.193.254:44547)
0810 14:58:18.392467 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.392469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.393591 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 14:58:18.393670 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.393671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.393680 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.393743 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.394627 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.395364 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396513 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.396789 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.396822 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.398365 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.398367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.398367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.398374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.399932 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.400277 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.400278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.400279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.400329 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.400511 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.400512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.400512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.400631 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.400712 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.401016 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.401085 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 14:58:18.402559 6601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:18.404160 6601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:18.404794 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: NON_VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: true } } }
I20260810 14:58:18.404887 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:18.406157 7478 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: attempting to promote NON_VOTER b8741d68b81a4a08ac8db88bd79de424 to VOTER
I20260810 14:58:18.406278 7478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:18.406772 7557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:58:18.406956 7479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 14:58:18.407045 7849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 LEARNER]: Refusing update from remote peer 139086d48a2f416e870b433ec4c427cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 14:58:18.407351 7479 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 14:58:18.407709 7780 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.407831 7557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408226 7849 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408596 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408668 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:18.408718 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408764 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:18.408793 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408825 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:18.408812 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.408897 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:58:18.409262 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "139086d48a2f416e870b433ec4c427cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:18.409353 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:58:18.413789 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.409645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45807 (local address 127.0.0.1:37556)
0810 14:58:18.409912 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.409949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.409999 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.410778 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.410779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.410781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.410825 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.410826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.411804 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.411805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.412854 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.412857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.413536 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.413537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.413596 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.413600 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.413653 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.413693 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260810 14:58:18.413851 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.409806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.28.193.193:45807)
0810 14:58:18.410595 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.410600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.410640 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:58:18.410682 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.410684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.410699 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.410771 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.410883 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.411712 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.412938 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.413455 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.413539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.413597 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.413669 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.413708 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":105,"threads_started":1}
I20260810 14:58:18.413892 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.409766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46199 (local address 127.0.0.1:58488)
0810 14:58:18.409917 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.409929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.409973 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.410488 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.410489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.410491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.410555 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.410557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.411544 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.411545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.412674 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.412677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.413672 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.413673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.413750 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.413753 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.413795 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.413828 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260810 14:58:18.414741 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.410216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:42045 (local address 127.0.0.1:49822)
0810 14:58:18.410337 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.410345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.410409 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.411684 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.411685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.411688 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.411731 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.411732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.412534 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.412535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.413631 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.413634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.414596 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.414598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.414641 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.414644 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.414671 (+ 27us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.414696 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260810 14:58:18.414789 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.410335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.28.193.195:42045)
0810 14:58:18.410539 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.410544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.410553 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:18.411562 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.411569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.411591 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.411663 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.411808 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.412456 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.413735 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.414508 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.414603 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.414651 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.414681 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.414714 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260810 14:58:18.414736 8019 tablet_service.cc:1467] Processing CreateTablet for tablet 2387938b525c47ffb4fd05c740f2f775 (DEFAULT_TABLE table=test-table [id=ee07a70b54d145489fe7734585615bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:18.415143 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2387938b525c47ffb4fd05c740f2f775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:18.415380 8450 tablet_service.cc:1467] Processing CreateTablet for tablet 2387938b525c47ffb4fd05c740f2f775 (DEFAULT_TABLE table=test-table [id=ee07a70b54d145489fe7734585615bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:18.415684 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2387938b525c47ffb4fd05c740f2f775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:18.417984 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.410020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.28.193.194:46199)
0810 14:58:18.410171 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.410176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.410184 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:18.410216 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.410218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.410230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.410302 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.410615 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.411473 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.412773 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.413506 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.413590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.413640 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.417835 (+ 4195us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.417888 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260810 14:58:18.418308 8597 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Bootstrap starting.
I20260810 14:58:18.418322 8596 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Bootstrap starting.
I20260810 14:58:18.418769 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 2387938b525c47ffb4fd05c740f2f775 (DEFAULT_TABLE table=test-table [id=ee07a70b54d145489fe7734585615bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:18.418974 8597 tablet_bootstrap.cc:654] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:18.418979 8596 tablet_bootstrap.cc:654] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:18.419083 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2387938b525c47ffb4fd05c740f2f775. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:18.419286 8597 log.cc:824] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:18.419286 8596 log.cc:824] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:18.420012 8596 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: No bootstrap required, opened a new log
I20260810 14:58:18.420012 8597 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: No bootstrap required, opened a new log
I20260810 14:58:18.420071 8597 ts_tablet_manager.cc:1397] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:18.420071 8596 ts_tablet_manager.cc:1397] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:18.421595 8600 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Bootstrap starting.
I20260810 14:58:18.421978 8597 raft_consensus.cc:348] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.422144 8600 tablet_bootstrap.cc:654] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:18.422155 8597 raft_consensus.cc:374] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:18.422189 8597 raft_consensus.cc:729] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1442114b594142b9112d03e7bdbc7c, State: Initialized, Role: FOLLOWER
I20260810 14:58:18.422302 8597 consensus_queue.cc:260] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.422429 8600 log.cc:824] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:18.422570 8597 ts_tablet_manager.cc:1428] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 14:58:18.422797 8555 heartbeater.cc:503] Master 127.28.193.254:44547 was elected leader, sending a full tablet report...
I20260810 14:58:18.422751 8596 raft_consensus.cc:348] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.422907 8596 raft_consensus.cc:374] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:18.422950 8596 raft_consensus.cc:729] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd04edbe6bfc499d8abdcaa12a8b8442, State: Initialized, Role: FOLLOWER
I20260810 14:58:18.423065 8596 consensus_queue.cc:260] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.423316 8596 ts_tablet_manager.cc:1428] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 14:58:18.423419 8121 heartbeater.cc:503] Master 127.28.193.254:44547 was elected leader, sending a full tablet report...
I20260810 14:58:18.424772 8600 tablet_bootstrap.cc:492] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: No bootstrap required, opened a new log
I20260810 14:58:18.424832 8600 ts_tablet_manager.cc:1397] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 14:58:18.426195 8600 raft_consensus.cc:348] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.426316 8600 raft_consensus.cc:374] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:18.426345 8600 raft_consensus.cc:729] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605ef25bc3ea4c84855d440bbfd6903d, State: Initialized, Role: FOLLOWER
I20260810 14:58:18.426451 8600 consensus_queue.cc:260] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.426667 8600 ts_tablet_manager.cc:1428] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:58:18.426786 8343 heartbeater.cc:503] Master 127.28.193.254:44547 was elected leader, sending a full tablet report...
I20260810 14:58:18.473294 8602 raft_consensus.cc:482] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:18.473439 8602 raft_consensus.cc:504] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.474040 8602 leader_election.cc:288] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807), 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
I20260810 14:58:18.477782 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.474278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:47973 (local address 127.28.193.193:45807)
0810 14:58:18.474333 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.474335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.474345 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:18.474404 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.474406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.474416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.474479 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.474647 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.475427 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.476833 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.477397 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.477481 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.477491 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.477643 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.477684 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:18.477784 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.474193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45807 (local address 127.28.193.195:47973)
0810 14:58:18.474251 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.474267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.474331 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.474495 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.474496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.474498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.474570 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.474572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.475513 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.475514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.476731 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.476734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.477483 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.477484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.477575 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.477577 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.477629 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.477670 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":64}
I20260810 14:58:18.477810 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.474390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:53679 (local address 127.28.193.194:46199)
0810 14:58:18.474444 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.474446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.474463 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:58:18.474488 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.474489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.474498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.474554 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:18.474675 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.475452 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.476833 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.477396 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.477481 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.477491 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.477643 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.477684 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:58:18.477799 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.474314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46199 (local address 127.28.193.195:53679)
0810 14:58:18.474369 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:18.474380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:18.474421 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:18.474562 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:18.474563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:18.474564 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:18.474616 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.474617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.475517 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.475518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.476731 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:18.476734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.477483 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.477484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:18.477571 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.477575 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:18.477629 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:58:18.477670 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":11}
I20260810 14:58:18.478348 8264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2387938b525c47ffb4fd05c740f2f775" candidate_uuid: "6b1442114b594142b9112d03e7bdbc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605ef25bc3ea4c84855d440bbfd6903d" is_pre_election: true
I20260810 14:58:18.478397 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2387938b525c47ffb4fd05c740f2f775" candidate_uuid: "6b1442114b594142b9112d03e7bdbc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" is_pre_election: true
I20260810 14:58:18.478542 8264 raft_consensus.cc:2454] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1442114b594142b9112d03e7bdbc7c in term 0.
I20260810 14:58:18.478542 8052 raft_consensus.cc:2454] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1442114b594142b9112d03e7bdbc7c in term 0.
I20260810 14:58:18.478780 8381 leader_election.cc:302] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605ef25bc3ea4c84855d440bbfd6903d, 6b1442114b594142b9112d03e7bdbc7c; no voters:
I20260810 14:58:18.478904 8602 raft_consensus.cc:2781] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:18.478974 8602 raft_consensus.cc:482] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:18.479002 8602 raft_consensus.cc:3037] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:18.479812 8602 raft_consensus.cc:504] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.479961 8602 leader_election.cc:288] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [CANDIDATE]: Term 1 election: Requested vote from peers fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807), 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199)
I20260810 14:58:18.480132 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2387938b525c47ffb4fd05c740f2f775" candidate_uuid: "6b1442114b594142b9112d03e7bdbc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
I20260810 14:58:18.480166 8264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2387938b525c47ffb4fd05c740f2f775" candidate_uuid: "6b1442114b594142b9112d03e7bdbc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "605ef25bc3ea4c84855d440bbfd6903d"
I20260810 14:58:18.480216 8052 raft_consensus.cc:3037] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:18.480249 8264 raft_consensus.cc:3037] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:18.480913 8052 raft_consensus.cc:2454] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1442114b594142b9112d03e7bdbc7c in term 1.
I20260810 14:58:18.481058 8264 raft_consensus.cc:2454] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1442114b594142b9112d03e7bdbc7c in term 1.
I20260810 14:58:18.481081 8381 leader_election.cc:302] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1442114b594142b9112d03e7bdbc7c, fd04edbe6bfc499d8abdcaa12a8b8442; no voters:
I20260810 14:58:18.481163 8602 raft_consensus.cc:2781] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:18.481328 8602 raft_consensus.cc:686] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 1 LEADER]: Becoming Leader. State: Replica: 6b1442114b594142b9112d03e7bdbc7c, State: Running, Role: LEADER
I20260810 14:58:18.481423 8602 consensus_queue.cc:237] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } }
I20260810 14:58:18.482370 6602 catalog_manager.cc:5495] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195). New cstate: current_term: 1 leader_uuid: "6b1442114b594142b9112d03e7bdbc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } health_report { overall_health: UNKNOWN } } }
W20260810 14:58:18.490684 8347 tablet.cc:2369] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:18.503489 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.500980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.28.193.195:42045)
0810 14:58:18.501051 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.501053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.501063 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:18.501105 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.501106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.501124 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.501195 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:18.501329 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.501910 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.502718 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.502955 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.503016 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.503228 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:18.503333 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:18.503397 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.503409 (+ 12us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.503430 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:58:18.505452 8264 raft_consensus.cc:1261] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Refusing update from remote peer 6b1442114b594142b9112d03e7bdbc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:18.505468 8052 raft_consensus.cc:1261] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Refusing update from remote peer 6b1442114b594142b9112d03e7bdbc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:18.505831 8602 consensus_queue.cc:1035] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:18.505986 8610 consensus_queue.cc:1035] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:18.508994 8619 mvcc.cc:205] Tried to move back new op lower bound from 7316987488275509248 to 7316987488180563968. Current Snapshot: MvccSnapshot[applied={T|T < 7316987488275509248}]
I20260810 14:58:18.511416 6601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:18.511495 8618 mvcc.cc:205] Tried to move back new op lower bound from 7316987488275509248 to 7316987488180563968. Current Snapshot: MvccSnapshot[applied={T|T < 7316987488275509248 or (T in {7316987488275509248})}]
I20260810 14:58:18.511519 6601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:18.512979 6601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 139086d48a2f416e870b433ec4c427cc: Sending DeleteTablet for 3 replicas of tablet 2387938b525c47ffb4fd05c740f2f775
I20260810 14:58:18.513484 8019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2387938b525c47ffb4fd05c740f2f775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:18 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:37556
I20260810 14:58:18.513559 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet 2387938b525c47ffb4fd05c740f2f775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:18 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:49822
I20260810 14:58:18.513803 8636 tablet_replica.cc:330] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: stopping tablet replica
I20260810 14:58:18.513785 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 2387938b525c47ffb4fd05c740f2f775 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:18 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:58488
I20260810 14:58:18.513896 8636 raft_consensus.cc:2229] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:18.513994 8636 raft_consensus.cc:2258] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:18.514007 8639 tablet_replica.cc:330] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: stopping tablet replica
I20260810 14:58:18.514103 8639 raft_consensus.cc:2229] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:18.514462 8637 tablet_replica.cc:330] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: stopping tablet replica
I20260810 14:58:18.514464 8636 ts_tablet_manager.cc:1905] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:18.514689 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 6547
I20260810 14:58:18.536665 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.514448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.28.193.252:44605)
0810 14:58:18.514494 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.514496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.525887 (+ 11391us) server_negotiation.cc:407] Connection header received
0810 14:58:18.526063 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.526065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.526082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.526127 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.527206 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.528207 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.529600 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.529983 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.530046 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.532234 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.532236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.532237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.532248 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.534859 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.535429 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.535430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.535431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.535513 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.535815 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.535816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.535817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.535991 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.536083 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.536432 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.536541 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:18.540959 8636 ts_tablet_manager.cc:1918] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:18.542253 8636 log.cc:1196] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/2387938b525c47ffb4fd05c740f2f775
I20260810 14:58:18.542577 8636 ts_tablet_manager.cc:1939] T 2387938b525c47ffb4fd05c740f2f775 P fd04edbe6bfc499d8abdcaa12a8b8442: Deleting consensus metadata
W20260810 14:58:18.544967 7941 connection.cc:535] server connection from 127.0.0.1:37556 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:58:18.545635 8637 raft_consensus.cc:2229] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:18.545850 8637 raft_consensus.cc:2258] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:18.546070 8639 raft_consensus.cc:2258] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:18.546715 8639 ts_tablet_manager.cc:1905] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:18.549106 8639 ts_tablet_manager.cc:1918] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:18.549204 8639 log.cc:1196] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/2387938b525c47ffb4fd05c740f2f775
I20260810 14:58:18.549521 8639 ts_tablet_manager.cc:1939] T 2387938b525c47ffb4fd05c740f2f775 P 605ef25bc3ea4c84855d440bbfd6903d: Deleting consensus metadata
W20260810 14:58:18.549970 8161 connection.cc:441] server connection from 127.0.0.1:58488 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58488 (request call id 1) could send its response
I20260810 14:58:18.550150 8637 ts_tablet_manager.cc:1905] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:18.551936 8637 ts_tablet_manager.cc:1918] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:18.552085 8637 log.cc:1196] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/2387938b525c47ffb4fd05c740f2f775
I20260810 14:58:18.552469 8637 ts_tablet_manager.cc:1939] T 2387938b525c47ffb4fd05c740f2f775 P 6b1442114b594142b9112d03e7bdbc7c: Deleting consensus metadata
W20260810 14:58:18.552834 8379 connection.cc:441] server connection from 127.0.0.1:49822 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49822 (request call id 1) could send its response
I20260810 14:58:18.555528 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.546757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.28.193.253:44487)
0810 14:58:18.546824 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.546826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.547035 (+ 209us) server_negotiation.cc:407] Connection header received
0810 14:58:18.547132 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.547133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.547144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.547173 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.548433 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.549169 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.550292 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.550562 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.550607 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.552370 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.552371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.552372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.552381 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.554215 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.554599 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.554600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.554600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.554665 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.554900 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.554901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.554902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.555020 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.555113 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.555348 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.555428 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 14:58:18.556910 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:18.547074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.28.193.252:44605)
0810 14:58:18.547118 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:18.547119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:18.548527 (+ 1408us) server_negotiation.cc:407] Connection header received
0810 14:58:18.548640 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:18.548641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:18.548661 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:18.548694 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:18.550528 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.551514 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.552478 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:18.552746 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:18.552794 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:18.554332 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:18.554333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:18.554333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:18.554342 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:18.555881 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.556231 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.556232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.556233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.556300 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:18.556457 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:18.556457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:18.556458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:18.556580 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:18.556673 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:18.556737 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 14:58:18.556816 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 14:58:19.511147 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:19.510831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:54335)
0810 14:58:19.511017 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:19.511099 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260810 14:58:19.511170 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:19.510915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:54021)
0810 14:58:19.511038 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:19.511118 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260810 14:58:19.511237 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:19.511047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:50285)
0810 14:58:19.511145 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:19.511200 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
W20260810 14:58:19.511336 8555 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
W20260810 14:58:19.511354 8121 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
W20260810 14:58:19.511381 8343 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44547 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.013921 8655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 139086d48a2f416e870b433ec4c427cc)
I20260810 14:58:20.013999 8657 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 139086d48a2f416e870b433ec4c427cc)
I20260810 14:58:20.014089 8655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:20.014127 8657 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:20.014365 8655 leader_election.cc:288] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605)
I20260810 14:58:20.014393 8657 leader_election.cc:288] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253:44487)
I20260810 14:58:20.014730 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54376)
0810 14:58:20.014638 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.014689 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":44,"threads_started":1}
W20260810 14:58:20.014863 7812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.014932 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54370)
0810 14:58:20.014726 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.014860 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":53,"threads_started":1}
W20260810 14:58:20.015197 7515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.015261 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.015181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54388)
0810 14:58:20.015218 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.015239 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 14:58:20.015332 7812 leader_election.cc:334] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.015700 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.015556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54404)
0810 14:58:20.015652 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.015675 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
W20260810 14:58:20.015762 7515 leader_election.cc:334] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.018427 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.28.193.253:44487)
0810 14:58:20.014930 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:20.014932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:20.014939 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:20.014966 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:20.014967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:20.014977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:20.015029 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:20.015210 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.015987 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.017447 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.018034 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.018108 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:20.018116 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:20.018306 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 14:58:20.018346 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":35,"threads_started":1}
I20260810 14:58:20.018431 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.28.193.252:44605)
0810 14:58:20.014883 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:20.014885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:20.015026 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:58:20.015047 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:20.015047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:20.015055 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:20.015102 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:20.015238 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.015993 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.017447 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.018034 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.018106 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:20.018116 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:20.018287 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 14:58:20.018347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260810 14:58:20.018491 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44487 (local address 127.0.0.1:39868)
0810 14:58:20.014765 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.014776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:20.014837 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:20.015082 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:20.015083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:20.015085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:20.015143 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:20.015145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.016099 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.016100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:20.017352 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:20.017354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.018122 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.018123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:20.018218 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:20.018222 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:20.018288 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 14:58:20.018365 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":22,"thread_start_us":49,"threads_started":1}
I20260810 14:58:20.018436 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.014540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:44605 (local address 127.0.0.1:43430)
0810 14:58:20.014924 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.014933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:20.014977 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:20.015132 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:20.015132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:20.015134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:20.015172 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:20.015175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.016123 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.016125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:20.017349 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:20.017353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:20.018122 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:20.018123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:20.018218 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:20.018222 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:20.018276 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 14:58:20.018321 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":67,"threads_started":1}
I20260810 14:58:20.018724 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8741d68b81a4a08ac8db88bd79de424" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" is_pre_election: true
I20260810 14:58:20.018822 7557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8741d68b81a4a08ac8db88bd79de424 in term 1.
I20260810 14:58:20.018922 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8741d68b81a4a08ac8db88bd79de424" is_pre_election: true
I20260810 14:58:20.018987 7813 leader_election.cc:302] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2547cca3d9af436b9e5bc4164c20f7d1, b8741d68b81a4a08ac8db88bd79de424; no voters: 139086d48a2f416e870b433ec4c427cc
I20260810 14:58:20.019017 7849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2547cca3d9af436b9e5bc4164c20f7d1 in term 1.
I20260810 14:58:20.019100 8657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:58:20.019131 8657 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 139086d48a2f416e870b433ec4c427cc)
I20260810 14:58:20.019150 8657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:58:20.019155 7516 leader_election.cc:302] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2547cca3d9af436b9e5bc4164c20f7d1, b8741d68b81a4a08ac8db88bd79de424; no voters: 139086d48a2f416e870b433ec4c427cc
I20260810 14:58:20.019254 8655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:58:20.019299 8655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 139086d48a2f416e870b433ec4c427cc)
I20260810 14:58:20.019317 8655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:58:20.020076 8655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:20.020076 8657 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:20.020220 8655 leader_election.cc:288] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 election: Requested vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605)
I20260810 14:58:20.020218 8657 leader_election.cc:288] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 election: Requested vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), 2547cca3d9af436b9e5bc4164c20f7d1 (127.28.193.253:44487)
I20260810 14:58:20.020358 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8741d68b81a4a08ac8db88bd79de424"
I20260810 14:58:20.020442 7849 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2547cca3d9af436b9e5bc4164c20f7d1 in current term 2: Already voted for candidate b8741d68b81a4a08ac8db88bd79de424 in this term.
I20260810 14:58:20.020416 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8741d68b81a4a08ac8db88bd79de424" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2547cca3d9af436b9e5bc4164c20f7d1"
I20260810 14:58:20.020473 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.020371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54422)
0810 14:58:20.020425 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.020451 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:58:20.020488 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.020294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54410)
0810 14:58:20.020409 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.020439 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 14:58:20.020498 7557 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b8741d68b81a4a08ac8db88bd79de424 in current term 2: Already voted for candidate 2547cca3d9af436b9e5bc4164c20f7d1 in this term.
I20260810 14:58:20.020742 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.020667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54424)
0810 14:58:20.020694 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.020721 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 14:58:20.020907 7812 leader_election.cc:334] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.020958 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.020862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54440)
0810 14:58:20.020910 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.020937 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 14:58:20.020962 7812 leader_election.cc:302] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8741d68b81a4a08ac8db88bd79de424; no voters: 139086d48a2f416e870b433ec4c427cc, 2547cca3d9af436b9e5bc4164c20f7d1
I20260810 14:58:20.021025 8657 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 14:58:20.021059 7515 leader_election.cc:334] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:20.021096 7515 leader_election.cc:302] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2547cca3d9af436b9e5bc4164c20f7d1; no voters: 139086d48a2f416e870b433ec4c427cc, b8741d68b81a4a08ac8db88bd79de424
I20260810 14:58:20.021154 8655 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 14:58:20.511874 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.511644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:49345)
0810 14:58:20.511796 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.511834 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260810 14:58:20.511871 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.511640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:34257)
0810 14:58:20.511787 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.511824 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260810 14:58:20.512094 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:20.511644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:33567)
0810 14:58:20.512025 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:20.512058 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":70,"threads_started":1}
I20260810 14:58:21.512575 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.512336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:52973)
0810 14:58:21.512477 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.512517 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260810 14:58:21.512588 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.512331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:39909)
0810 14:58:21.512472 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.512537 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260810 14:58:21.512737 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.512464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:48705)
0810 14:58:21.512668 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.512701 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":58,"threads_started":1}
I20260810 14:58:21.752007 8684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:21.752135 8684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:21.752323 8684 leader_election.cc:288] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605)
I20260810 14:58:21.752574 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8741d68b81a4a08ac8db88bd79de424" is_pre_election: true
I20260810 14:58:21.752664 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.752466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54692)
0810 14:58:21.752571 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.752621 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260810 14:58:21.752712 7849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2547cca3d9af436b9e5bc4164c20f7d1 in term 2.
I20260810 14:58:21.752864 7516 leader_election.cc:302] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2547cca3d9af436b9e5bc4164c20f7d1, b8741d68b81a4a08ac8db88bd79de424; no voters:
I20260810 14:58:21.752983 8684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 14:58:21.753037 8684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:21.753052 8684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:58:21.753082 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.752970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54694)
0810 14:58:21.753023 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.753055 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 14:58:21.753180 7515 leader_election.cc:334] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:21.753856 8684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:21.753978 8684 leader_election.cc:288] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 election: Requested vote from peers 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547), b8741d68b81a4a08ac8db88bd79de424 (127.28.193.252:44605)
I20260810 14:58:21.754186 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.754069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54696)
0810 14:58:21.754144 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.754168 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:58:21.754149 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8741d68b81a4a08ac8db88bd79de424"
I20260810 14:58:21.754225 7849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:58:21.754535 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.754423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54702)
0810 14:58:21.754469 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.754494 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 14:58:21.754607 7515 leader_election.cc:334] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
I20260810 14:58:21.754985 7849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2547cca3d9af436b9e5bc4164c20f7d1 in term 3.
I20260810 14:58:21.755146 7516 leader_election.cc:302] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2547cca3d9af436b9e5bc4164c20f7d1, b8741d68b81a4a08ac8db88bd79de424; no voters: 139086d48a2f416e870b433ec4c427cc
I20260810 14:58:21.755316 8684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:58:21.755439 8684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [term 3 LEADER]: Becoming Leader. State: Replica: 2547cca3d9af436b9e5bc4164c20f7d1, State: Running, Role: LEADER
I20260810 14:58:21.755514 8684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } }
I20260810 14:58:21.755833 8692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2547cca3d9af436b9e5bc4164c20f7d1. Latest consensus state: current_term: 3 leader_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:21.755927 8692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:21.756124 8696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:21.756526 8696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:21.756588 8696 catalog_manager.cc:1259] Loaded cluster ID: 217a14952eb94220ba367531acc804f5
I20260810 14:58:21.756618 8696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:21.756798 8696 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:21.756866 8696 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Loaded TSK: 0
I20260810 14:58:21.757009 8696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:21.791854 7542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:21.793612 7542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:21.794735 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.794619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54728)
0810 14:58:21.794669 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.794706 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 14:58:21.794725 7849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [term 3 FOLLOWER]: Refusing update from remote peer 2547cca3d9af436b9e5bc4164c20f7d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 14:58:21.794953 8692 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 14:58:21.795068 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.794984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.0.0.1:54732)
0810 14:58:21.795013 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.795045 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260810 14:58:21.795125 7515 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 -> Peer 139086d48a2f416e870b433ec4c427cc (127.28.193.254:44547): Couldn't send request to peer 139086d48a2f416e870b433ec4c427cc. Status: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:58:21.795534 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2547cca3d9af436b9e5bc4164c20f7d1. Latest consensus state: current_term: 3 leader_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:21.795627 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:58:21.796074 8684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:21.796167 8684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:21.796180 8698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "139086d48a2f416e870b433ec4c427cc" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44547 } } peers { permanent_uuid: "2547cca3d9af436b9e5bc4164c20f7d1" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44487 } attrs { promote: false } } peers { permanent_uuid: "b8741d68b81a4a08ac8db88bd79de424" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 44605 } attrs { promote: false } } }
I20260810 14:58:21.796265 8698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8741d68b81a4a08ac8db88bd79de424 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:58:21.796473 8700 mvcc.cc:205] Tried to move back new op lower bound from 7316987501749129216 to 7316987501591109632. Current Snapshot: MvccSnapshot[applied={T|T < 7316987501749129216 or (T in {7316987501749129216})}]
I20260810 14:58:21.800871 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.797998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46199 (local address 127.0.0.1:58498)
0810 14:58:21.798129 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.798141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:21.798194 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:21.798539 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:21.798541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:21.798543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:21.798597 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.798599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799318 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.799319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.800085 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.800087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.800656 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.800658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.800719 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.800724 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:58:21.800755 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 14:58:21.800794 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260810 14:58:21.801256 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.798247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.28.193.194:46199)
0810 14:58:21.798362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.798364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.798372 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:21.798400 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.798401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.798411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.798667 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:21.798712 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799195 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.800169 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.800567 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.801120 (+ 553us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.801131 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.801150 (+ 19us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.801190 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260810 14:58:21.801592 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.797895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45807 (local address 127.0.0.1:37560)
0810 14:58:21.797921 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.797931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:21.797980 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:21.798260 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:21.798261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:21.798263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:21.798320 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.798322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799275 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.799276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.800460 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.800463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.801323 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.801324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.801377 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.801379 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:21.801437 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 14:58:21.801509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13,"mutex_wait_us":19}
I20260810 14:58:21.801622 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 487d1595932244219145e769d2ab2fae (DEFAULT_TABLE table=test-table [id=fbf68fe6a0764b9c931d19eafc76d5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:21.801661 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.798002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:42045 (local address 127.0.0.1:49836)
0810 14:58:21.798341 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.798350 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:21.798394 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:21.798578 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:21.798578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:21.798580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:21.798620 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.798622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799446 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.799447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.800515 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.800517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.801303 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.801304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.801369 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.801373 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:21.801435 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 14:58:21.801509 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260810 14:58:21.801744 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.798193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.28.193.195:42045)
0810 14:58:21.798297 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.798299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.798437 (+ 138us) server_negotiation.cc:407] Connection header received
0810 14:58:21.798463 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.798464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.798488 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.798538 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:21.798686 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799375 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.800615 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.801170 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.801255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.801264 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.801609 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.801658 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260810 14:58:21.801780 8232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487d1595932244219145e769d2ab2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:21.801551 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.797981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.28.193.193:45807)
0810 14:58:21.798116 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.798118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.798127 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:21.798154 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.798154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.798165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.798384 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:21.798439 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.799188 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.800554 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.801225 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.801321 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.801330 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.801446 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.801483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260810 14:58:21.801955 8450 tablet_service.cc:1467] Processing CreateTablet for tablet 487d1595932244219145e769d2ab2fae (DEFAULT_TABLE table=test-table [id=fbf68fe6a0764b9c931d19eafc76d5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:21.802060 8450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487d1595932244219145e769d2ab2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:21.802042 8019 tablet_service.cc:1467] Processing CreateTablet for tablet 487d1595932244219145e769d2ab2fae (DEFAULT_TABLE table=test-table [id=fbf68fe6a0764b9c931d19eafc76d5a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:21.802184 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 487d1595932244219145e769d2ab2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:21.803349 8717 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Bootstrap starting.
I20260810 14:58:21.803622 8718 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Bootstrap starting.
I20260810 14:58:21.803833 8717 tablet_bootstrap.cc:654] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:21.803999 8718 tablet_bootstrap.cc:654] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:21.804142 8719 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: Bootstrap starting.
I20260810 14:58:21.804450 8717 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: No bootstrap required, opened a new log
I20260810 14:58:21.804507 8717 ts_tablet_manager.cc:1397] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:21.804546 8719 tablet_bootstrap.cc:654] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:21.804598 8718 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: No bootstrap required, opened a new log
I20260810 14:58:21.804649 8718 ts_tablet_manager.cc:1397] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:21.804646 8717 raft_consensus.cc:348] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.804739 8717 raft_consensus.cc:374] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:21.804772 8717 raft_consensus.cc:729] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605ef25bc3ea4c84855d440bbfd6903d, State: Initialized, Role: FOLLOWER
I20260810 14:58:21.804762 8718 raft_consensus.cc:348] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.804829 8718 raft_consensus.cc:374] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:21.804852 8718 raft_consensus.cc:729] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1442114b594142b9112d03e7bdbc7c, State: Initialized, Role: FOLLOWER
I20260810 14:58:21.804865 8717 consensus_queue.cc:260] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.804914 8718 consensus_queue.cc:260] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.805090 8719 tablet_bootstrap.cc:492] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: No bootstrap required, opened a new log
I20260810 14:58:21.805073 8717 ts_tablet_manager.cc:1428] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:21.805147 8719 ts_tablet_manager.cc:1397] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:21.805085 8718 ts_tablet_manager.cc:1428] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:21.805258 8719 raft_consensus.cc:348] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.805327 8719 raft_consensus.cc:374] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:21.805348 8719 raft_consensus.cc:729] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd04edbe6bfc499d8abdcaa12a8b8442, State: Initialized, Role: FOLLOWER
I20260810 14:58:21.805354 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.805247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:38245)
0810 14:58:21.805304 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.805331 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 14:58:21.805392 8719 consensus_queue.cc:260] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.805555 8719 ts_tablet_manager.cc:1428] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:21.805676 8346 heartbeater.cc:503] Master 127.28.193.253:44487 was elected leader, sending a full tablet report...
I20260810 14:58:21.805783 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.805647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:49947)
0810 14:58:21.805738 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.805763 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:58:21.806154 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.805957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:38197)
0810 14:58:21.806105 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.806128 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260810 14:58:21.806154 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.806057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:49191)
0810 14:58:21.806103 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.806128 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260810 14:58:21.806510 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.806431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:41097)
0810 14:58:21.806468 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.806488 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 14:58:21.806710 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.806631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:47161)
0810 14:58:21.806667 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.806687 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 14:58:21.806797 8122 heartbeater.cc:503] Master 127.28.193.253:44487 was elected leader, sending a full tablet report...
I20260810 14:58:21.806869 8559 heartbeater.cc:503] Master 127.28.193.253:44487 was elected leader, sending a full tablet report...
W20260810 14:58:21.819456 8123 tablet.cc:2369] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:58:21.883606 8562 tablet.cc:2369] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:21.933403 8727 raft_consensus.cc:482] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:21.933529 8727 raft_consensus.cc:504] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.933945 8727 leader_election.cc:288] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807), 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:21.936854 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.934214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:50975 (local address 127.28.193.193:45807)
0810 14:58:21.934271 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.934273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.934498 (+ 225us) server_negotiation.cc:407] Connection header received
0810 14:58:21.934553 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.934554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.934564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.934610 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:21.934738 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.935198 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936140 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.936546 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936619 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.936630 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.936749 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.936787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:21.936862 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.934268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:35817 (local address 127.28.193.195:42045)
0810 14:58:21.934321 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.934323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.934333 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:21.934361 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.934362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.934372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.934439 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:21.934630 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.935113 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936157 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.936529 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936602 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.936613 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.936752 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.936794 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:58:21.936874 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.934140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:42045 (local address 127.28.193.194:35817)
0810 14:58:21.934226 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.934238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:21.934296 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:21.934478 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:21.934480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:21.934483 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:21.934554 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.934556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.935195 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.935196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.936055 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.936057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.936608 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.936679 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.936683 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:21.936744 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:58:21.936784 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":43}
I20260810 14:58:21.936877 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.934141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45807 (local address 127.28.193.194:50975)
0810 14:58:21.934431 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.934445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:21.934494 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:21.934618 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:21.934618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:21.934620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:21.934682 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.934684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.935271 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.935272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.936055 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:21.936058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.936623 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.936625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:21.936688 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.936692 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:21.936738 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:58:21.936776 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":104,"thread_start_us":128,"threads_started":1}
I20260810 14:58:21.937186 8482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487d1595932244219145e769d2ab2fae" candidate_uuid: "605ef25bc3ea4c84855d440bbfd6903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1442114b594142b9112d03e7bdbc7c" is_pre_election: true
I20260810 14:58:21.937358 8482 raft_consensus.cc:2454] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 605ef25bc3ea4c84855d440bbfd6903d in term 0.
I20260810 14:58:21.937355 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487d1595932244219145e769d2ab2fae" candidate_uuid: "605ef25bc3ea4c84855d440bbfd6903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" is_pre_election: true
I20260810 14:58:21.937440 8052 raft_consensus.cc:2454] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 605ef25bc3ea4c84855d440bbfd6903d in term 0.
I20260810 14:58:21.937573 8162 leader_election.cc:302] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605ef25bc3ea4c84855d440bbfd6903d, 6b1442114b594142b9112d03e7bdbc7c; no voters:
I20260810 14:58:21.937749 8727 raft_consensus.cc:2781] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:21.937808 8727 raft_consensus.cc:482] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:21.937831 8727 raft_consensus.cc:3037] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:21.938562 8727 raft_consensus.cc:504] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.938787 8727 leader_election.cc:288] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [CANDIDATE]: Term 1 election: Requested vote from peers fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807), 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045)
I20260810 14:58:21.938900 8482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487d1595932244219145e769d2ab2fae" candidate_uuid: "605ef25bc3ea4c84855d440bbfd6903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1442114b594142b9112d03e7bdbc7c"
I20260810 14:58:21.938930 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "487d1595932244219145e769d2ab2fae" candidate_uuid: "605ef25bc3ea4c84855d440bbfd6903d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442"
I20260810 14:58:21.938987 8482 raft_consensus.cc:3037] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:21.939005 8052 raft_consensus.cc:3037] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:21.939661 8052 raft_consensus.cc:2454] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605ef25bc3ea4c84855d440bbfd6903d in term 1.
I20260810 14:58:21.939663 8482 raft_consensus.cc:2454] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 605ef25bc3ea4c84855d440bbfd6903d in term 1.
I20260810 14:58:21.939821 8162 leader_election.cc:302] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 605ef25bc3ea4c84855d440bbfd6903d, 6b1442114b594142b9112d03e7bdbc7c; no voters:
I20260810 14:58:21.939939 8727 raft_consensus.cc:2781] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:21.940107 8727 raft_consensus.cc:686] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 1 LEADER]: Becoming Leader. State: Replica: 605ef25bc3ea4c84855d440bbfd6903d, State: Running, Role: LEADER
I20260810 14:58:21.940238 8727 consensus_queue.cc:237] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } }
I20260810 14:58:21.940460 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.940348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:59221)
0810 14:58:21.940387 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.940427 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 14:58:21.941071 7542 catalog_manager.cc:5495] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d reported cstate change: term changed from 0 to 1, leader changed from <none> to 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194). New cstate: current_term: 1 leader_uuid: "605ef25bc3ea4c84855d440bbfd6903d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "605ef25bc3ea4c84855d440bbfd6903d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 } health_report { overall_health: UNKNOWN } } }
I20260810 14:58:21.981865 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.979702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58504 (local address 127.28.193.194:46199)
0810 14:58:21.979752 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:21.979754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:21.979763 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:21.979806 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:21.979806 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:21.979826 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:21.979899 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:21.980015 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.980478 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.981176 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:21.981382 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:21.981446 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:21.981548 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:21.981650 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:21.981702 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:21.981776 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:58:21.981801 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 14:58:21.982980 8052 raft_consensus.cc:1261] T 487d1595932244219145e769d2ab2fae P fd04edbe6bfc499d8abdcaa12a8b8442 [term 1 FOLLOWER]: Refusing update from remote peer 605ef25bc3ea4c84855d440bbfd6903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:21.983078 8482 raft_consensus.cc:1261] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Refusing update from remote peer 605ef25bc3ea4c84855d440bbfd6903d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:21.983230 8727 consensus_queue.cc:1035] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd04edbe6bfc499d8abdcaa12a8b8442" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:21.983372 8732 consensus_queue.cc:1035] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1442114b594142b9112d03e7bdbc7c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 42045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:21.984128 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.984003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:37959)
0810 14:58:21.984066 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.984097 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:58:21.984241 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.984148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:36063)
0810 14:58:21.984182 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.984211 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 14:58:21.985177 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.985074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:50919)
0810 14:58:21.985118 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.985149 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 14:58:21.985451 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.985371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.194:33595)
0810 14:58:21.985408 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.985429 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 14:58:21.985631 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.985477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.193:38967)
0810 14:58:21.985574 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.985603 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 14:58:21.986240 8741 mvcc.cc:205] Tried to move back new op lower bound from 7316987502520184832 to 7316987502347980800. Current Snapshot: MvccSnapshot[applied={T|T < 7316987502520184832}]
I20260810 14:58:21.987301 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:21.987190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44547 (local address 127.28.193.195:38651)
0810 14:58:21.987239 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:21.987273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 14:58:21.987344 7541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:21.987433 7541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:21.988672 7541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2547cca3d9af436b9e5bc4164c20f7d1: Sending DeleteTablet for 3 replicas of tablet 487d1595932244219145e769d2ab2fae
I20260810 14:58:21.988965 8232 tablet_service.cc:1514] Processing DeleteTablet for tablet 487d1595932244219145e769d2ab2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:21 UTC) from {username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:58498
I20260810 14:58:21.988974 8450 tablet_service.cc:1514] Processing DeleteTablet for tablet 487d1595932244219145e769d2ab2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:21 UTC) from {username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:49836
I20260810 14:58:21.988986 8019 tablet_service.cc:1514] Processing DeleteTablet for tablet 487d1595932244219145e769d2ab2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:21 UTC) from {username='oryx', principal='oryx/127.28.193.253@KRBTEST.COM'} at 127.0.0.1:37560
I20260810 14:58:21.989285 8754 tablet_replica.cc:330] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: stopping tablet replica
I20260810 14:58:21.989365 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 7773
I20260810 14:58:21.989439 8756 tablet_replica.cc:330] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: stopping tablet replica
I20260810 14:58:22.001821 8756 raft_consensus.cc:2229] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:22.001924 8756 raft_consensus.cc:2258] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:22.002135 8754 raft_consensus.cc:2229] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:22.002265 8754 raft_consensus.cc:2258] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:22.002390 8756 ts_tablet_manager.cc:1905] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:22.002609 8754 ts_tablet_manager.cc:1905] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:22.004396 8756 ts_tablet_manager.cc:1918] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:22.004396 8754 ts_tablet_manager.cc:1918] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:22.004465 8754 log.cc:1196] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/487d1595932244219145e769d2ab2fae
I20260810 14:58:22.004465 8756 log.cc:1196] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/487d1595932244219145e769d2ab2fae
I20260810 14:58:22.004705 8756 ts_tablet_manager.cc:1939] T 487d1595932244219145e769d2ab2fae P 6b1442114b594142b9112d03e7bdbc7c: Deleting consensus metadata
I20260810 14:58:22.004705 8754 ts_tablet_manager.cc:1939] T 487d1595932244219145e769d2ab2fae P 605ef25bc3ea4c84855d440bbfd6903d: Deleting consensus metadata
I20260810 14:58:22.005048 7519 catalog_manager.cc:4841] TS 6b1442114b594142b9112d03e7bdbc7c (127.28.193.195:42045): tablet 487d1595932244219145e769d2ab2fae (table test-table [id=fbf68fe6a0764b9c931d19eafc76d5a4]) successfully deleted
I20260810 14:58:22.005048 7520 catalog_manager.cc:4841] TS 605ef25bc3ea4c84855d440bbfd6903d (127.28.193.194:46199): tablet 487d1595932244219145e769d2ab2fae (table test-table [id=fbf68fe6a0764b9c931d19eafc76d5a4]) successfully deleted
I20260810 14:58:22.005729 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:22.005613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45807 (local address 127.0.0.1:37562)
0810 14:58:22.005651 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:22.005677 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 14:58:22.005811 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 8134
W20260810 14:58:22.005882 7520 catalog_manager.cc:4584] TS fd04edbe6bfc499d8abdcaa12a8b8442 (127.28.193.193:45807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 487d1595932244219145e769d2ab2fae: Network error: Client connection negotiation failed: client connection to 127.28.193.193:45807: connect: Connection refused (error 111)
I20260810 14:58:22.018865 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 8351
I20260810 14:58:22.033558 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 7336
I20260810 14:58:22.044025 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 7624
2026-08-10T14:58:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6562 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:22 dist-test-slave-cvwc krb5kdc[8763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:22 dist-test-slave-cvwc krb5kdc[8763](info): set up 2 sockets
Aug 10 14:58:22 dist-test-slave-cvwc krb5kdc[8763](info): commencing operation
krb5kdc: starting...
W20260810 14:58:23.225989 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab.
Entry for principal oryx/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab.
I20260810 14:58:23.348546 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34703
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab
--principal=oryx/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:23.431239 8785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:23.431435 8785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:23.431460 8785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:23.432536 8785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:23.432569 8785 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:58:23.432580 8785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:23.432591 8785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:23.432603 8785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:23.432616 8785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:23.434150 8785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34703
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.254.keytab
--principal=oryx/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:23.434342 8785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:23.434512 8785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:23.439177 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.439184 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.439186 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:23.439302 8785 server_base.cc:1034] running on GCE node
I20260810 14:58:23.439736 8785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:23.440014 8785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:23.441164 8785 hybrid_clock.cc:648] HybridClock initialized: now 1786373903441155 us; error 33 us; skew 500 ppm
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:23.450304 8785 init.cc:377] Logged in from keytab as oryx/127.28.193.254@KRBTEST.COM (short username oryx)
I20260810 14:58:23.450754 8785 webserver.cc:460] Webserver started at http://127.28.193.254:35949/ using document root <none> and password file <none>
I20260810 14:58:23.450985 8785 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:23.451035 8785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:23.451167 8785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:23.452208 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "f127d43dad3449cca285bd8ff3d6de9b"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "9250e4bf22198b26f7a23b290ce60979"
server_key_iv: "c335f445715dfd1e51e1528abec21dab"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.452564 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "f127d43dad3449cca285bd8ff3d6de9b"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "9250e4bf22198b26f7a23b290ce60979"
server_key_iv: "c335f445715dfd1e51e1528abec21dab"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.454005 8785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:23.454758 8811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.454953 8785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:23.455042 8785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "f127d43dad3449cca285bd8ff3d6de9b"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "9250e4bf22198b26f7a23b290ce60979"
server_key_iv: "c335f445715dfd1e51e1528abec21dab"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.455104 8785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:23.493129 8785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:23.495196 8785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:23.495352 8785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:23.499992 8785 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34703
I20260810 14:58:23.500056 8914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34703 every 8 connection(s)
I20260810 14:58:23.500380 8785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:23.501040 8916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:23.503347 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Bootstrap starting.
I20260810 14:58:23.503926 8916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:23.503993 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 8785
I20260810 14:58:23.504088 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:23.504258 29447 external_mini_cluster.cc:1428] Setting key b87ace950833a10cdd88110326cc2353
I20260810 14:58:23.504241 8916 log.cc:824] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:23.504943 8916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: No bootstrap required, opened a new log
I20260810 14:58:23.507084 8916 raft_consensus.cc:348] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } }
I20260810 14:58:23.507232 8916 raft_consensus.cc:374] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:23.507265 8916 raft_consensus.cc:729] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f127d43dad3449cca285bd8ff3d6de9b, State: Initialized, Role: FOLLOWER
I20260810 14:58:23.507407 8916 consensus_queue.cc:260] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } }
I20260810 14:58:23.507488 8916 raft_consensus.cc:388] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:23.507517 8916 raft_consensus.cc:482] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:23.507565 8916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:23.508046 8916 raft_consensus.cc:504] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } }
I20260810 14:58:23.508141 8916 leader_election.cc:302] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f127d43dad3449cca285bd8ff3d6de9b; no voters:
I20260810 14:58:23.508294 8916 leader_election.cc:288] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:23.508342 8925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:23.508530 8916 sys_catalog.cc:564] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:23.508529 8925 raft_consensus.cc:686] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [term 1 LEADER]: Becoming Leader. State: Replica: f127d43dad3449cca285bd8ff3d6de9b, State: Running, Role: LEADER
I20260810 14:58:23.509142 8925 consensus_queue.cc:237] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } }
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:23.510918 8928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f127d43dad3449cca285bd8ff3d6de9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } } }
I20260810 14:58:23.511034 8928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:23.510915 8926 sys_catalog.cc:455] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f127d43dad3449cca285bd8ff3d6de9b. Latest consensus state: current_term: 1 leader_uuid: "f127d43dad3449cca285bd8ff3d6de9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f127d43dad3449cca285bd8ff3d6de9b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34703 } } }
I20260810 14:58:23.511236 8926 sys_catalog.cc:458] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b [sys.catalog]: This master's current role is: LEADER
W20260810 14:58:23.511339 8950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:58:23.511387 8950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:58:23.511439 8954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:23.511595 8954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:23.514791 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.504716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.28.193.254:34703)
0810 14:58:23.504901 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:23.504911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:23.504942 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:58:23.505383 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:23.505391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:23.505441 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:23.505570 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:23.506239 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.507140 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.507998 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.508547 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.508659 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.511344 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:23.511348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:23.511348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:23.511360 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:23.513185 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.513525 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.513527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.513528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.513644 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.513896 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.513897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.513897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.514204 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:23.514363 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:23.514444 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 14:58:23.514579 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260810 14:58:23.518115 8954 catalog_manager.cc:1347] Generated new cluster ID: acc8bf8c2fe747828c3beba8096b32af
I20260810 14:58:23.518174 8954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:23.542282 8954 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:23.542847 8954 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:23.546231 8954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Generated new TSK 0
I20260810 14:58:23.546419 8954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
I20260810 14:58:23.576529 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34703
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:23.659463 8963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:23.659680 8963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:23.659714 8963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:23.660916 8963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:23.660961 8963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:23.660988 8963 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:58:23.661012 8963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:23.662606 8963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:23.662858 8963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:23.663093 8963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:23.668196 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.668210 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.668227 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:23.668393 8963 server_base.cc:1034] running on GCE node
I20260810 14:58:23.668617 8963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:23.669013 8963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:23.670190 8963 hybrid_clock.cc:648] HybridClock initialized: now 1786373903670165 us; error 45 us; skew 500 ppm
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:23.672767 8963 init.cc:377] Logged in from keytab as oryx/127.28.193.193@KRBTEST.COM (short username oryx)
I20260810 14:58:23.673218 8963 webserver.cc:460] Webserver started at http://127.28.193.193:37181/ using document root <none> and password file <none>
I20260810 14:58:23.673451 8963 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:23.673501 8963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:23.673629 8963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:23.674722 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "d937773821df41bcb9f3401ab3174ec9"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "be6fe0c531e3122922b86ec9e159f55c"
server_key_iv: "e7e7f5f7c1597b6d39a814be9126304d"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.675060 8963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "d937773821df41bcb9f3401ab3174ec9"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "be6fe0c531e3122922b86ec9e159f55c"
server_key_iv: "e7e7f5f7c1597b6d39a814be9126304d"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.676308 8963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 14:58:23.677024 8989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.677182 8963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:23.677265 8963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "d937773821df41bcb9f3401ab3174ec9"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "be6fe0c531e3122922b86ec9e159f55c"
server_key_iv: "e7e7f5f7c1597b6d39a814be9126304d"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.677330 8963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:23.687100 8963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:23.689116 8963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:23.689249 8963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:23.689780 8963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:23.689822 8963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.689849 8963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:23.689880 8963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:23.696761 8963 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:38239
I20260810 14:58:23.696878 9168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:38239 every 8 connection(s)
I20260810 14:58:23.697204 8963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:23.699079 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.690176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54479 (local address 127.28.193.254:34703)
0810 14:58:23.690244 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:23.690246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:23.690431 (+ 185us) server_negotiation.cc:407] Connection header received
0810 14:58:23.691067 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:23.691068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:23.691079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:23.691143 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:23.691989 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.692537 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.694005 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.694234 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.694300 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.696377 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:23.696378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:23.696378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:23.696385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:23.697901 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.698232 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.698233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.698233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.698280 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.698557 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.698558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.698558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.698654 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:23.698698 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:23.698948 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:58:23.699016 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:58:23.699167 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.690133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.193:54479)
0810 14:58:23.690292 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:23.690341 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:23.690967 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:23.691227 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:23.691231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:23.691576 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:23.691863 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.691881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.692610 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.692611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.693915 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.693919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.694347 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.694348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.694460 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.694921 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:23.694929 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:23.696281 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:23.697967 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.697972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.697974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.698172 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.698368 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.698369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.698369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.698498 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.698702 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:23.698705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:23.698811 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 14:58:23.698897 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:58:23.699016 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260810 14:58:23.699666 9173 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34703
I20260810 14:58:23.699743 9173 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:23.699952 9173 heartbeater.cc:511] Master 127.28.193.254:34703 requested a full tablet report, sending...
I20260810 14:58:23.700502 8843 ts_manager.cc:194] Registered new tserver with Master: d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239)
I20260810 14:58:23.701154 8843 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:54479
I20260810 14:58:23.702237 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 8963
I20260810 14:58:23.702329 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:23.702534 29447 external_mini_cluster.cc:1428] Setting key 9445caef1bc93803089244e3cb73df76
I20260810 14:58:23.706151 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.700838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:46159 (local address 127.28.193.254:34703)
0810 14:58:23.700887 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:23.700889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:23.700895 (+ 6us) server_negotiation.cc:407] Connection header received
0810 14:58:23.700913 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:23.700914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:23.700922 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:23.700966 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:23.701256 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.701898 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.702657 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.702892 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.702963 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.703809 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:23.703810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:23.703810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:23.703816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:23.705082 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.705410 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.705411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.705412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.705466 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.705644 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.705644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.705645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.705787 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:23.705859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:23.705995 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 14:58:23.706061 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:58:23.706166 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.700663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.193:46159)
0810 14:58:23.700791 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:23.700801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:23.700847 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:23.700968 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:23.700969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:23.701138 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:23.701184 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.701186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.701986 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.701987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.702552 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.702554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.702964 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.702965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.703012 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.703398 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:23.703404 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:23.703711 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:23.705166 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.705167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.705168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.705330 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.705529 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.705529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.705530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.705587 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.705851 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:23.705852 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:23.705899 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:58:23.705962 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 14:58:23.706062 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
I20260810 14:58:23.728106 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34703
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:23.809322 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:23.809535 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:23.809566 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:23.810793 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:23.810840 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:23.810868 9184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:58:23.810890 9184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:23.812422 9184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:23.812666 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:23.812899 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:23.818035 9184 server_base.cc:1034] running on GCE node
W20260810 14:58:23.818014 9192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.817999 9191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.818014 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:23.818392 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:23.818658 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:23.819821 9184 hybrid_clock.cc:648] HybridClock initialized: now 1786373903819797 us; error 37 us; skew 500 ppm
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:23.822304 9184 init.cc:377] Logged in from keytab as oryx/127.28.193.194@KRBTEST.COM (short username oryx)
I20260810 14:58:23.822747 9184 webserver.cc:460] Webserver started at http://127.28.193.194:38355/ using document root <none> and password file <none>
I20260810 14:58:23.823021 9184 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:23.823074 9184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:23.823199 9184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:23.824249 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "c30c47cd298d4534bf41ad9b0061c76d"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "a328b489481785c41fa0d211acad5299"
server_key_iv: "a5f1c00444c73e3ebc37edfaf81b815a"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.824594 9184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "c30c47cd298d4534bf41ad9b0061c76d"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "a328b489481785c41fa0d211acad5299"
server_key_iv: "a5f1c00444c73e3ebc37edfaf81b815a"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.825937 9184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:23.826637 9206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.826807 9184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:23.826892 9184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "c30c47cd298d4534bf41ad9b0061c76d"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "a328b489481785c41fa0d211acad5299"
server_key_iv: "a5f1c00444c73e3ebc37edfaf81b815a"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.826953 9184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:23.841833 9184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:23.843772 9184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:23.843907 9184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:23.844398 9184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:23.844442 9184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.844472 9184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:23.844488 9184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:23.850977 9184 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:36745
I20260810 14:58:23.851038 9387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:36745 every 8 connection(s)
I20260810 14:58:23.851366 9184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:23.853493 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 9184
I20260810 14:58:23.853600 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:23.853839 29447 external_mini_cluster.cc:1428] Setting key 89029ea3623dafee358af83b868778b3
I20260810 14:58:23.854008 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.844886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:45147 (local address 127.28.193.254:34703)
0810 14:58:23.844957 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:23.844959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:23.845100 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:58:23.845652 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:23.845653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:23.845665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:23.845746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:23.846541 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.847284 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.848706 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.848998 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.849081 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.851539 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:23.851540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:23.851540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:23.851549 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:23.852660 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.853043 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.853045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.853045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.853105 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.853335 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.853335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.853335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.853440 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:23.853488 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:23.853830 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 14:58:23.853910 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:58:23.854192 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.844826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.194:45147)
0810 14:58:23.844970 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:23.845009 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:23.845544 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:23.845826 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:23.845830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:23.846153 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:23.846427 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.846434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.847388 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.847389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.848609 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.848612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.849098 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.849100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.849219 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.849741 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:23.849751 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:23.851431 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:23.852735 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.852740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.852742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.852946 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.853169 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.853170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.853170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.853264 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.853502 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:23.853509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:23.853672 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 14:58:23.853807 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 14:58:23.853971 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260810 14:58:23.854697 9388 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34703
I20260810 14:58:23.854794 9388 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:23.854975 9388 heartbeater.cc:511] Master 127.28.193.254:34703 requested a full tablet report, sending...
I20260810 14:58:23.855342 8843 ts_manager.cc:194] Registered new tserver with Master: c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:23.855751 8843 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:45147
I20260810 14:58:23.861625 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.856062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:40275 (local address 127.28.193.254:34703)
0810 14:58:23.856106 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:23.856108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:23.856205 (+ 97us) server_negotiation.cc:407] Connection header received
0810 14:58:23.856312 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:23.856313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:23.856324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:23.856352 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:23.856721 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.857273 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.858055 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.858302 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.858356 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.859244 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:23.859245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:23.859245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:23.859251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:23.860543 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.860919 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.860920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.860920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.860975 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:23.861165 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:23.861166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:23.861166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:23.861295 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:23.861342 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:23.861507 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 14:58:23.861564 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:23.861688 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:23.856006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.194:40275)
0810 14:58:23.856146 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:23.856156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:23.856230 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:23.856407 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:23.856407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:23.856602 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:23.856660 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.856661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.857391 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.857392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.857990 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:23.857993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:23.858377 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:23.858378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:23.858440 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:23.858820 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:23.858825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:23.859154 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:23.860632 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.860634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.860634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.860849 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.861039 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:23.861040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:23.861040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:23.861107 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:23.861363 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:23.861364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:23.861412 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:58:23.861487 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:58:23.861595 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
I20260810 14:58:23.880852 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34703
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:39979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:23.963606 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:23.963819 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:23.963851 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:23.965041 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:23.965086 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:23.965114 9404 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:58:23.965137 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:23.966743 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.193.254:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:23.966989 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:23.967206 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:23.972311 9404 server_base.cc:1034] running on GCE node
W20260810 14:58:23.972324 9415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.972437 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:23.972323 9417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:23.972755 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:23.972966 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:23.974107 9404 hybrid_clock.cc:648] HybridClock initialized: now 1786373903974095 us; error 27 us; skew 500 ppm
Aug 10 14:58:23 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:23.976568 9404 init.cc:377] Logged in from keytab as oryx/127.28.193.195@KRBTEST.COM (short username oryx)
I20260810 14:58:23.976979 9404 webserver.cc:460] Webserver started at http://127.28.193.195:37455/ using document root <none> and password file <none>
I20260810 14:58:23.977195 9404 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:23.977243 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:23.977375 9404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:23.978485 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "dba47596ada34b1cacc7837020cca7d2"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "e27fa7b9812737194dc51697580df0eb"
server_key_iv: "1ee6df9ada9f5ec983a14a2495af67f0"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.978830 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "dba47596ada34b1cacc7837020cca7d2"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "e27fa7b9812737194dc51697580df0eb"
server_key_iv: "1ee6df9ada9f5ec983a14a2495af67f0"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.980118 9404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:23.980983 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:23.981143 9404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:23.981220 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "dba47596ada34b1cacc7837020cca7d2"
format_stamp: "Formatted at 2026-08-10 14:58:23 on dist-test-slave-cvwc"
server_key: "e27fa7b9812737194dc51697580df0eb"
server_key_iv: "1ee6df9ada9f5ec983a14a2495af67f0"
server_key_version: "encryptionkey@0"
I20260810 14:58:23.981284 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:24.001422 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:24.003546 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:24.003688 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:24.004204 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:24.004245 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.004274 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:24.004289 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:24.010957 9404 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:36405
I20260810 14:58:24.011066 9620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:36405 every 8 connection(s)
I20260810 14:58:24.011376 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:24.013646 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.004681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:33263 (local address 127.28.193.254:34703)
0810 14:58:24.004730 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.004732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.004979 (+ 247us) server_negotiation.cc:407] Connection header received
0810 14:58:24.005571 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.005572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.005586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.005655 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.006333 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.006882 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.007906 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.008149 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.008213 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.011279 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:24.011281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:24.011281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:24.011289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:24.012579 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.012947 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.012947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.012947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.012994 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.013159 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.013160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.013160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.013253 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:24.013296 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.013517 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.013584 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 14:58:24.013819 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.004694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.195:33263)
0810 14:58:24.004850 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.004899 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.005420 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.005780 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.005785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.006044 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:24.006227 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.006233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.006972 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.006973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.007783 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.007785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.008232 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.008233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.008356 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.008875 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:24.008885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:24.011168 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:24.012649 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.012655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.012657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.012884 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.013032 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.013032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.013032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.013117 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.013299 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:24.013302 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:24.013413 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.013489 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.013606 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":100,"threads_started":1}
I20260810 14:58:24.014298 9621 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34703
I20260810 14:58:24.014407 9621 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:24.014587 9621 heartbeater.cc:511] Master 127.28.193.254:34703 requested a full tablet report, sending...
I20260810 14:58:24.014966 8842 ts_manager.cc:194] Registered new tserver with Master: dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405)
I20260810 14:58:24.015398 8842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:33263
I20260810 14:58:24.016052 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 9404
I20260810 14:58:24.016120 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:24.016252 29447 external_mini_cluster.cc:1428] Setting key c8558d93ab0d1d3367ef3cbd7227dac1
I20260810 14:58:24.017059 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:58:24.020696 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.015266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.28.193.195:40635)
0810 14:58:24.015385 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.015396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.015459 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.015580 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.015581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.015766 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:24.015824 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.015826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.016524 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.016525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.017257 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.017260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.017568 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.017569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.017633 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.017995 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:24.017999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:24.018317 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:24.019876 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.019877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.019877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.020021 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.020204 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.020205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.020205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.020249 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.020483 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:24.020484 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:24.020516 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.020559 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.020629 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260810 14:58:24.020699 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.015353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40635 (local address 127.28.193.254:34703)
0810 14:58:24.015391 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.015393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.015442 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:58:24.015518 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.015519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.015529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.015576 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.015921 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.016441 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.017320 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.017496 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.017548 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.018406 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:24.018407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:24.018408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:24.018414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:24.019790 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.020089 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.020089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.020090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.020149 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.020304 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.020304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.020305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.020426 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:24.020492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.020585 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.020643 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:58:24.038488 8923 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:24.033787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.28.193.254:34703)
0810 14:58:24.033843 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.033845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.033856 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:24.033896 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.033897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.033905 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.033951 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.034465 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.034931 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.035599 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.035764 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.035805 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.038410 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45838: BlockingRecv error: recv got EOF from 127.0.0.1:45838 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373904, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:24.045305 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.039229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.28.193.254:34703)
0810 14:58:24.039295 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.039296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.039304 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:24.039370 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.039371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.039379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.039426 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.039840 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.040289 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.040933 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.041118 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.041172 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.043155 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:24.043157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:24.043157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:24.043164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:24.044313 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.044618 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.044619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.044619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.044669 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.044841 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.044842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.044842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.044943 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:24.045030 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.045152 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.045219 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:58:24.046864 8842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:24.048373 8842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:24.054558 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38239 (local address 127.0.0.1:41162)
0810 14:58:24.050946 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.050988 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.051070 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.051268 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.051272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.051274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.051336 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.051338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052282 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.052283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.053319 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.053321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054257 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.054347 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054362 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.054430 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.054484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":82,"threads_started":1}
I20260810 14:58:24.054647 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.28.193.193:38239)
0810 14:58:24.050985 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.050990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.051052 (+ 62us) server_negotiation.cc:407] Connection header received
0810 14:58:24.051157 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.051160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.051175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.051248 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.051419 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052194 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.053429 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054172 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054257 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054321 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.054469 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.054512 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260810 14:58:24.054951 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.28.193.195:36405)
0810 14:58:24.050987 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.050990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.051215 (+ 225us) server_negotiation.cc:407] Connection header received
0810 14:58:24.051262 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.051264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.051278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.051452 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.051546 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052662 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.053787 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054454 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054541 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054598 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.054793 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.054836 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
I20260810 14:58:24.054870 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.0.0.1:50680)
0810 14:58:24.051106 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.051116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.051162 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.051469 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.051470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.051471 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.051495 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.051496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052775 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.052777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.053685 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.053687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054635 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.054697 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054701 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.054754 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.054796 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":33,"thread_start_us":50,"threads_started":1}
I20260810 14:58:24.055225 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.28.193.194:36745)
0810 14:58:24.051216 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.051220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.051229 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:24.051262 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.051264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.051278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.051358 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.051637 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052699 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.053986 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054722 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054799 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054861 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.055085 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.055127 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
I20260810 14:58:24.055424 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.050822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36745 (local address 127.0.0.1:47678)
0810 14:58:24.051030 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.051040 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.051094 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.051376 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.051377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.051379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.051409 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.051411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.052788 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.052789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.053897 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.053900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.054897 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.054898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.054953 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.054957 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.055003 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.055360 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":53,"threads_started":1}
I20260810 14:58:24.055794 9067 tablet_service.cc:1467] Processing CreateTablet for tablet 848556d0307148d393ba0c4093090a5a (DEFAULT_TABLE table=test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.055797 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 848556d0307148d393ba0c4093090a5a (DEFAULT_TABLE table=test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.056077 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 848556d0307148d393ba0c4093090a5a (DEFAULT_TABLE table=test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.056178 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848556d0307148d393ba0c4093090a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.056178 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848556d0307148d393ba0c4093090a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.056340 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848556d0307148d393ba0c4093090a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.059659 9664 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Bootstrap starting.
I20260810 14:58:24.059754 9666 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Bootstrap starting.
I20260810 14:58:24.060372 9667 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Bootstrap starting.
I20260810 14:58:24.060431 9664 tablet_bootstrap.cc:654] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.060431 9666 tablet_bootstrap.cc:654] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.060753 9666 log.cc:824] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:24.060753 9664 log.cc:824] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:24.061010 9667 tablet_bootstrap.cc:654] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.061334 9667 log.cc:824] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:24.061534 9666 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: No bootstrap required, opened a new log
I20260810 14:58:24.061607 9666 ts_tablet_manager.cc:1397] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:24.062054 9664 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: No bootstrap required, opened a new log
I20260810 14:58:24.062108 9667 tablet_bootstrap.cc:492] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: No bootstrap required, opened a new log
I20260810 14:58:24.062178 9667 ts_tablet_manager.cc:1397] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:24.062179 9664 ts_tablet_manager.cc:1397] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:58:24.063478 9667 raft_consensus.cc:348] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.063612 9667 raft_consensus.cc:374] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.063633 9667 raft_consensus.cc:729] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.063561 9666 raft_consensus.cc:348] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.063724 9666 raft_consensus.cc:374] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.063767 9666 raft_consensus.cc:729] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.063771 9667 consensus_queue.cc:260] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.063887 9666 consensus_queue.cc:260] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.064036 9667 ts_tablet_manager.cc:1428] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:58:24.064153 9666 ts_tablet_manager.cc:1428] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 14:58:24.064158 9173 heartbeater.cc:503] Master 127.28.193.254:34703 was elected leader, sending a full tablet report...
I20260810 14:58:24.064201 9621 heartbeater.cc:503] Master 127.28.193.254:34703 was elected leader, sending a full tablet report...
I20260810 14:58:24.064440 9664 raft_consensus.cc:348] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.064590 9664 raft_consensus.cc:374] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.064631 9664 raft_consensus.cc:729] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.064729 9664 consensus_queue.cc:260] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.064968 9664 ts_tablet_manager.cc:1428] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:58:24.064980 9388 heartbeater.cc:503] Master 127.28.193.254:34703 was elected leader, sending a full tablet report...
W20260810 14:58:24.101935 9391 tablet.cc:2369] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:58:24.197990 9174 tablet.cc:2369] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:24.238519 9676 raft_consensus.cc:482] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:24.238656 9676 raft_consensus.cc:504] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.239043 9676 leader_election.cc:288] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405), d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239)
I20260810 14:58:24.242892 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.239320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:40973 (local address 127.28.193.193:38239)
0810 14:58:24.239413 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.239415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.239570 (+ 155us) server_negotiation.cc:407] Connection header received
0810 14:58:24.239601 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.239602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.239610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.239669 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.239831 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.240508 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.241939 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.242488 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.242569 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.242578 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.242808 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.242836 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:58:24.242960 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.239253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38239 (local address 127.28.193.194:40973)
0810 14:58:24.239498 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.239505 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.239559 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.239672 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.239673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.239684 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.239749 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.239751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.240577 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.240578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.241836 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.241840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.242607 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.242609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.242722 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.242727 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.242796 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.242843 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":20,"thread_start_us":128,"threads_started":1}
I20260810 14:58:24.243275 9226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.239246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.28.193.194:40389)
0810 14:58:24.239308 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.239319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.239397 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.239564 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.239565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.239567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.239642 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.239644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.240537 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.240538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.241836 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.241840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.242607 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.242609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.243115 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.243119 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.243168 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.243210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":44}
I20260810 14:58:24.243309 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.239310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:40389 (local address 127.28.193.195:36405)
0810 14:58:24.239374 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.239376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.239387 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:24.239481 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.239482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.239492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.239543 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.239715 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.240461 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.241940 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.242506 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.242568 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.242578 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.243207 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.243250 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:58:24.243345 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848556d0307148d393ba0c4093090a5a" candidate_uuid: "c30c47cd298d4534bf41ad9b0061c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9" is_pre_election: true
I20260810 14:58:24.243513 9098 raft_consensus.cc:2454] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30c47cd298d4534bf41ad9b0061c76d in term 0.
I20260810 14:58:24.243577 9543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848556d0307148d393ba0c4093090a5a" candidate_uuid: "c30c47cd298d4534bf41ad9b0061c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2" is_pre_election: true
I20260810 14:58:24.243723 9543 raft_consensus.cc:2454] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30c47cd298d4534bf41ad9b0061c76d in term 0.
I20260810 14:58:24.243759 9217 leader_election.cc:302] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, d937773821df41bcb9f3401ab3174ec9; no voters:
I20260810 14:58:24.243902 9676 raft_consensus.cc:2781] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:24.243954 9676 raft_consensus.cc:482] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:24.243979 9676 raft_consensus.cc:3037] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.244668 9676 raft_consensus.cc:504] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.244796 9676 leader_election.cc:288] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [CANDIDATE]: Term 1 election: Requested vote from peers dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405), d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239)
I20260810 14:58:24.244989 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848556d0307148d393ba0c4093090a5a" candidate_uuid: "c30c47cd298d4534bf41ad9b0061c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9"
I20260810 14:58:24.245055 9543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848556d0307148d393ba0c4093090a5a" candidate_uuid: "c30c47cd298d4534bf41ad9b0061c76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2"
I20260810 14:58:24.245087 9098 raft_consensus.cc:3037] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.245121 9543 raft_consensus.cc:3037] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.245955 9098 raft_consensus.cc:2454] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30c47cd298d4534bf41ad9b0061c76d in term 1.
I20260810 14:58:24.245977 9543 raft_consensus.cc:2454] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30c47cd298d4534bf41ad9b0061c76d in term 1.
I20260810 14:58:24.246143 9217 leader_election.cc:302] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, d937773821df41bcb9f3401ab3174ec9; no voters:
I20260810 14:58:24.246223 9676 raft_consensus.cc:2781] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:24.246376 9676 raft_consensus.cc:686] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 1 LEADER]: Becoming Leader. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Running, Role: LEADER
I20260810 14:58:24.246490 9676 consensus_queue.cc:237] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } }
I20260810 14:58:24.247347 8843 catalog_manager.cc:5495] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d reported cstate change: term changed from 0 to 1, leader changed from <none> to c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194). New cstate: current_term: 1 leader_uuid: "c30c47cd298d4534bf41ad9b0061c76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } health_report { overall_health: HEALTHY } } }
W20260810 14:58:24.262108 9622 tablet.cc:2369] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:24.279822 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.277476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.28.193.194:36745)
0810 14:58:24.277534 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.277537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.277546 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:24.277611 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.277613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.277632 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.277697 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.277872 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.278335 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.279098 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.279283 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.279339 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.279515 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.279616 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.279686 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.279727 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.279748 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 14:58:24.281832 9098 raft_consensus.cc:1261] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Refusing update from remote peer c30c47cd298d4534bf41ad9b0061c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.281839 9543 raft_consensus.cc:1261] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Refusing update from remote peer c30c47cd298d4534bf41ad9b0061c76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.282177 9676 consensus_queue.cc:1035] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.282316 9679 consensus_queue.cc:1035] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.285552 9685 mvcc.cc:205] Tried to move back new op lower bound from 7316987511935193088 to 7316987511794081792. Current Snapshot: MvccSnapshot[applied={T|T < 7316987511935193088 or (T in {7316987511935193088})}]
I20260810 14:58:24.286986 8843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.287067 8843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.287825 8843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Sending DeleteTablet for 3 replicas of tablet 848556d0307148d393ba0c4093090a5a
I20260810 14:58:24.288139 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet 848556d0307148d393ba0c4093090a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41162
I20260810 14:58:24.288151 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet 848556d0307148d393ba0c4093090a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 14:58:24.288151 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet 848556d0307148d393ba0c4093090a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260810 14:58:24.288447 9699 tablet_replica.cc:330] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: stopping tablet replica
I20260810 14:58:24.288681 8843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:58:24.288803 9701 tablet_replica.cc:330] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: stopping tablet replica
W20260810 14:58:24.288815 8843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:24.288894 9701 raft_consensus.cc:2229] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.289093 9699 raft_consensus.cc:2229] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.289208 9699 raft_consensus.cc:2258] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.289471 9701 raft_consensus.cc:2258] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.289831 9699 ts_tablet_manager.cc:1905] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.289975 9067 tablet_service.cc:1467] Processing CreateTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 (DEFAULT_TABLE table=test-table [id=a237980f7fde460bbd087227038214ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.290048 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 (DEFAULT_TABLE table=test-table [id=a237980f7fde460bbd087227038214ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.290093 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705ce1f1f7694c0b8199939f8a91df43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.290092 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 (DEFAULT_TABLE table=test-table [id=a237980f7fde460bbd087227038214ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.290169 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705ce1f1f7694c0b8199939f8a91df43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.290181 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705ce1f1f7694c0b8199939f8a91df43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.291422 9667 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Bootstrap starting.
I20260810 14:58:24.291422 9666 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Bootstrap starting.
I20260810 14:58:24.291821 9667 tablet_bootstrap.cc:654] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.291822 9666 tablet_bootstrap.cc:654] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.292138 9664 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Bootstrap starting.
I20260810 14:58:24.292462 9667 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: No bootstrap required, opened a new log
I20260810 14:58:24.292485 9666 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: No bootstrap required, opened a new log
I20260810 14:58:24.292506 9667 ts_tablet_manager.cc:1397] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:24.292527 9666 ts_tablet_manager.cc:1397] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:24.292528 9664 tablet_bootstrap.cc:654] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.292649 9667 raft_consensus.cc:348] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.292712 9667 raft_consensus.cc:374] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.292683 9666 raft_consensus.cc:348] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.292732 9667 raft_consensus.cc:729] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.292745 9666 raft_consensus.cc:374] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.292774 9666 raft_consensus.cc:729] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.292798 9667 consensus_queue.cc:260] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.292825 9666 consensus_queue.cc:260] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.292900 9667 ts_tablet_manager.cc:1428] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:24.292910 9666 ts_tablet_manager.cc:1428] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:24.293931 9701 ts_tablet_manager.cc:1905] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.294682 9699 ts_tablet_manager.cc:1918] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.294764 9699 log.cc:1196] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/848556d0307148d393ba0c4093090a5a
I20260810 14:58:24.295006 9699 ts_tablet_manager.cc:1939] T 848556d0307148d393ba0c4093090a5a P d937773821df41bcb9f3401ab3174ec9: Deleting consensus metadata
I20260810 14:58:24.295187 9702 tablet_replica.cc:330] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: stopping tablet replica
I20260810 14:58:24.295348 9702 raft_consensus.cc:2229] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:24.295406 8822 catalog_manager.cc:4841] TS d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239): tablet 848556d0307148d393ba0c4093090a5a (table test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]) successfully deleted
I20260810 14:58:24.295439 9702 raft_consensus.cc:2258] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.295221 9701 ts_tablet_manager.cc:1918] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.295751 9701 log.cc:1196] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/848556d0307148d393ba0c4093090a5a
I20260810 14:58:24.295867 9664 tablet_bootstrap.cc:492] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: No bootstrap required, opened a new log
I20260810 14:58:24.295920 9664 ts_tablet_manager.cc:1397] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 14:58:24.296012 9701 ts_tablet_manager.cc:1939] T 848556d0307148d393ba0c4093090a5a P dba47596ada34b1cacc7837020cca7d2: Deleting consensus metadata
I20260810 14:58:24.296051 9664 raft_consensus.cc:348] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.296121 9664 raft_consensus.cc:374] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.296146 9664 raft_consensus.cc:729] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.296186 9664 consensus_queue.cc:260] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.296268 9664 ts_tablet_manager.cc:1428] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:24.296417 8817 catalog_manager.cc:4841] TS dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405): tablet 848556d0307148d393ba0c4093090a5a (table test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]) successfully deleted
I20260810 14:58:24.296660 9702 ts_tablet_manager.cc:1905] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.297924 9702 ts_tablet_manager.cc:1918] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.297992 9702 log.cc:1196] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/848556d0307148d393ba0c4093090a5a
I20260810 14:58:24.298193 9702 ts_tablet_manager.cc:1939] T 848556d0307148d393ba0c4093090a5a P c30c47cd298d4534bf41ad9b0061c76d: Deleting consensus metadata
I20260810 14:58:24.298575 8822 catalog_manager.cc:4841] TS c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745): tablet 848556d0307148d393ba0c4093090a5a (table test-table [id=f3cb16c0a280485baf39fe85ccb7a64f]) successfully deleted
I20260810 14:58:24.362067 9672 raft_consensus.cc:482] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:24.362188 9672 raft_consensus.cc:504] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.362583 9672 leader_election.cc:288] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745), dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405)
I20260810 14:58:24.366024 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.362938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:44871 (local address 127.28.193.195:36405)
0810 14:58:24.362987 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.362989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.363141 (+ 152us) server_negotiation.cc:407] Connection header received
0810 14:58:24.363177 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.363178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.363188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.363232 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.363460 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.364172 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.365157 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.365582 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.365649 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.365658 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.365901 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.365947 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 14:58:24.366259 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.362809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.28.193.193:44871)
0810 14:58:24.363055 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.363065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.363118 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.363294 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.363296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.363298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.363374 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.363376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.364254 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.364255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.365052 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.365054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.365735 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.365737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.365816 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.365821 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.366123 (+ 302us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.366170 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":33,"thread_start_us":59,"threads_started":1}
I20260810 14:58:24.366451 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.363031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:36833 (local address 127.28.193.194:36745)
0810 14:58:24.363071 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.363073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.363082 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:24.363112 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.363113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.363124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.363209 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.363382 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.364124 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.365565 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.366099 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.366164 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.366172 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.366348 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.366390 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 14:58:24.366533 9543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705ce1f1f7694c0b8199939f8a91df43" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2" is_pre_election: true
I20260810 14:58:24.366644 9543 raft_consensus.cc:2454] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d937773821df41bcb9f3401ab3174ec9 in term 0.
I20260810 14:58:24.366648 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.362808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36745 (local address 127.28.193.193:36833)
0810 14:58:24.362938 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.362950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.363012 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.363214 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.363215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.363217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.363297 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.363300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.364224 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.364225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.365472 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.365476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.366202 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.366203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.366266 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.366270 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.366530 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.366574 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260810 14:58:24.366890 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705ce1f1f7694c0b8199939f8a91df43" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d" is_pre_election: true
I20260810 14:58:24.367002 9316 raft_consensus.cc:2454] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d937773821df41bcb9f3401ab3174ec9 in term 0.
I20260810 14:58:24.367089 8991 leader_election.cc:302] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d937773821df41bcb9f3401ab3174ec9, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.367252 9672 raft_consensus.cc:2781] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:24.367300 9672 raft_consensus.cc:482] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:24.367319 9672 raft_consensus.cc:3037] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.367983 9672 raft_consensus.cc:504] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.368111 9672 leader_election.cc:288] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 election: Requested vote from peers c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745), dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405)
I20260810 14:58:24.368367 9543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705ce1f1f7694c0b8199939f8a91df43" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2"
I20260810 14:58:24.368387 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705ce1f1f7694c0b8199939f8a91df43" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d"
I20260810 14:58:24.368448 9316 raft_consensus.cc:3037] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.368448 9543 raft_consensus.cc:3037] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.369196 9316 raft_consensus.cc:2454] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d937773821df41bcb9f3401ab3174ec9 in term 1.
I20260810 14:58:24.369196 9543 raft_consensus.cc:2454] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d937773821df41bcb9f3401ab3174ec9 in term 1.
I20260810 14:58:24.369361 8991 leader_election.cc:302] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d937773821df41bcb9f3401ab3174ec9, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.369462 9672 raft_consensus.cc:2781] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:24.369606 9672 raft_consensus.cc:686] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 1 LEADER]: Becoming Leader. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Running, Role: LEADER
I20260810 14:58:24.369670 9672 consensus_queue.cc:237] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.370465 8842 catalog_manager.cc:5495] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d937773821df41bcb9f3401ab3174ec9 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "d937773821df41bcb9f3401ab3174ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373903, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.254@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:24.409067 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.402561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.28.193.254:34703)
0810 14:58:24.402615 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.402617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.402727 (+ 110us) server_negotiation.cc:407] Connection header received
0810 14:58:24.402765 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.402766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.402777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.402851 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.403379 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.403867 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.404605 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.404795 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.404851 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.406892 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:24.406893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:24.406894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:24.406899 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:24.408086 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.408431 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.408432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.408432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.408476 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.408615 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.408615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.408615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.408703 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:24.408745 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.408862 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.408966 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":30}
I20260810 14:58:24.409076 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.402474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34703 (local address 127.0.0.1:45846)
0810 14:58:24.402646 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.402658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.402721 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.402900 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.402902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.403222 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:24.403269 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.403271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.403945 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.403946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.404476 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.404478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.404858 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.404858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.404895 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.405435 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:24.405440 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:24.406805 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:24.408166 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.408180 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.408182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.408375 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.408524 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:24.408525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:24.408525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:24.408575 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:24.408746 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:24.408750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:24.408791 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.408837 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.408923 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260810 14:58:24.411285 8842 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:45846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:58:24.411491 8842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:24.413168 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 993b58dfb0a4474da3b94426cfc89166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa72916458f4f12bd76a0bf68254b28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:58:24.413168 9505 tablet_service.cc:1467] Processing CreateTablet for tablet 993b58dfb0a4474da3b94426cfc89166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa72916458f4f12bd76a0bf68254b28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:58:24.413175 9067 tablet_service.cc:1467] Processing CreateTablet for tablet 993b58dfb0a4474da3b94426cfc89166 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5fa72916458f4f12bd76a0bf68254b28]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:58:24.413334 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 993b58dfb0a4474da3b94426cfc89166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.413335 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 993b58dfb0a4474da3b94426cfc89166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.413334 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 993b58dfb0a4474da3b94426cfc89166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.414597 9664 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: Bootstrap starting.
I20260810 14:58:24.414865 9667 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: Bootstrap starting.
I20260810 14:58:24.414852 9666 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: Bootstrap starting.
I20260810 14:58:24.414992 9664 tablet_bootstrap.cc:654] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.415253 9666 tablet_bootstrap.cc:654] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.415273 9667 tablet_bootstrap.cc:654] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.415870 9666 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: No bootstrap required, opened a new log
I20260810 14:58:24.415923 9666 ts_tablet_manager.cc:1397] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:24.415936 9667 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: No bootstrap required, opened a new log
I20260810 14:58:24.415979 9667 ts_tablet_manager.cc:1397] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:24.416075 9664 tablet_bootstrap.cc:492] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: No bootstrap required, opened a new log
I20260810 14:58:24.416124 9664 ts_tablet_manager.cc:1397] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:24.416090 9666 raft_consensus.cc:348] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416157 9666 raft_consensus.cc:374] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.416124 9667 raft_consensus.cc:348] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416185 9666 raft_consensus.cc:729] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.416188 9667 raft_consensus.cc:374] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.416206 9667 raft_consensus.cc:729] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.416224 9666 consensus_queue.cc:260] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416246 9667 consensus_queue.cc:260] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416234 9664 raft_consensus.cc:348] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416289 9664 raft_consensus.cc:374] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.416304 9666 ts_tablet_manager.cc:1428] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:24.416319 9664 raft_consensus.cc:729] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.416328 9667 ts_tablet_manager.cc:1428] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.416356 9664 consensus_queue.cc:260] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.416389 9672 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.416440 9664 ts_tablet_manager.cc:1428] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.416455 9672 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.416453 9673 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.416471 9691 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.416518 9673 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.416546 9691 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.544968 9673 raft_consensus.cc:482] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:24.545089 9673 raft_consensus.cc:504] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.545491 9673 leader_election.cc:288] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:24.548496 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.545999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:49457 (local address 127.28.193.194:36745)
0810 14:58:24.546078 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.546079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.546088 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:24.546116 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.546116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.546125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.546175 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.546351 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.546804 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.547799 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.548154 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.548217 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.548227 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.548388 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.548427 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:58:24.548511 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.545827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:51237 (local address 127.28.193.193:38239)
0810 14:58:24.545907 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.545909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.545943 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:58:24.546060 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.546061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.546071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.546096 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:24.546297 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.546790 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.547811 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.548154 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.548218 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.548227 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.548403 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.548446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:58:24.548522 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.545899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36745 (local address 127.28.193.195:49457)
0810 14:58:24.545991 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.546001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.546052 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.546195 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.546195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.546197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.546271 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.546274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.546895 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.546896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.547698 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.547700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.548241 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.548242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.548310 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.548314 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.548364 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.548406 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260810 14:58:24.548524 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.545691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38239 (local address 127.28.193.195:51237)
0810 14:58:24.545857 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.545871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.545948 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.546153 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.546154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.546157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:24.546224 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.546226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.546886 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.546887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.547708 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.547711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.548245 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.548246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.548323 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.548327 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.548380 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.548423 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":60,"thread_start_us":93,"threads_started":1}
I20260810 14:58:24.548790 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993b58dfb0a4474da3b94426cfc89166" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d" is_pre_election: true
I20260810 14:58:24.548887 9316 raft_consensus.cc:2454] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:24.548966 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993b58dfb0a4474da3b94426cfc89166" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9" is_pre_election: true
I20260810 14:58:24.549062 9098 raft_consensus.cc:2454] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:24.549072 9439 leader_election.cc:302] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.549203 9673 raft_consensus.cc:2781] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:24.549252 9673 raft_consensus.cc:482] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:24.549275 9673 raft_consensus.cc:3037] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.549988 9673 raft_consensus.cc:504] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.550097 9673 leader_election.cc:288] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Requested vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:24.550259 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993b58dfb0a4474da3b94426cfc89166" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9"
I20260810 14:58:24.550323 9098 raft_consensus.cc:3037] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.550290 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "993b58dfb0a4474da3b94426cfc89166" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d"
I20260810 14:58:24.550383 9316 raft_consensus.cc:3037] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.550923 9098 raft_consensus.cc:2454] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 1.
I20260810 14:58:24.551105 9439 leader_election.cc:302] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d937773821df41bcb9f3401ab3174ec9, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.551204 9673 raft_consensus.cc:2781] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:24.551204 9316 raft_consensus.cc:2454] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 1.
I20260810 14:58:24.551306 9673 raft_consensus.cc:686] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [term 1 LEADER]: Becoming Leader. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Running, Role: LEADER
I20260810 14:58:24.551378 9673 consensus_queue.cc:237] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.551741 9735 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: TxnStatusTablet state changed. Reason: New leader dba47596ada34b1cacc7837020cca7d2. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.551838 9735 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:58:24.552016 9737 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:58:24.552111 9737 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:58:24.552150 8843 catalog_manager.cc:5495] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba47596ada34b1cacc7837020cca7d2 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } health_report { overall_health: HEALTHY } } }
I20260810 14:58:24.563223 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.560727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.0.0.1:50692)
0810 14:58:24.560820 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.560831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.560881 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.561039 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.561039 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.561041 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:58:24.561094 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.561096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.561744 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.561745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.562332 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.562334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.562691 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.562692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.562774 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.562790 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.563042 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.563052 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.563092 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.563151 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:58:24.563225 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.560804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.28.193.195:36405)
0810 14:58:24.560885 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.560887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.560896 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:24.560960 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.560961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.560980 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.561035 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.561163 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.561636 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.562417 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.562621 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.562676 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.562879 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.562981 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.563043 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.563106 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.563126 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:58:24.564512 9098 raft_consensus.cc:1261] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.564616 9316 raft_consensus.cc:1261] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.564800 9735 consensus_queue.cc:1035] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.564985 9673 consensus_queue.cc:1035] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.565697 9691 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: TxnStatusTablet state changed. Reason: New leader dba47596ada34b1cacc7837020cca7d2. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.565738 9672 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: TxnStatusTablet state changed. Reason: New leader dba47596ada34b1cacc7837020cca7d2. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.565824 9691 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.565852 9672 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.566553 9673 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.566672 9673 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:58:24.566742 9691 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.566819 9691 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P c30c47cd298d4534bf41ad9b0061c76d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.567083 9673 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.567169 9673 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P dba47596ada34b1cacc7837020cca7d2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:58:24.567337 9672 tablet_replica.cc:439] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } } }
I20260810 14:58:24.567432 9672 tablet_replica.cc:442] T 993b58dfb0a4474da3b94426cfc89166 P d937773821df41bcb9f3401ab3174ec9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:58:24.571120 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.568416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.28.193.193:38239)
0810 14:58:24.568474 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.568475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.568483 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:24.568515 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.568516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.568536 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.568619 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.568711 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.569439 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.570234 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.570544 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.570614 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.570761 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.570884 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.570961 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.571006 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.571025 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 14:58:24.576810 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.573938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54597 (local address 127.28.193.195:36405)
0810 14:58:24.574000 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.574002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.574258 (+ 256us) server_negotiation.cc:407] Connection header received
0810 14:58:24.574293 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.574295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.574305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.574386 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.574519 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.575236 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.576036 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.576348 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.576416 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.576587 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.576624 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.576673 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.576716 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.576736 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:58:24.576838 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.573871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.28.193.193:54597)
0810 14:58:24.574074 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.574171 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.574248 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.574391 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.574392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.574393 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:58:24.574456 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.574458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.575335 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.575336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.575939 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.575947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.576432 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.576433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.576489 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.576505 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.576677 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.576678 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.576711 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.576746 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260810 14:58:24.578990 9543 raft_consensus.cc:1261] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Refusing update from remote peer d937773821df41bcb9f3401ab3174ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.578996 9316 raft_consensus.cc:1261] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Refusing update from remote peer d937773821df41bcb9f3401ab3174ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.579232 9672 consensus_queue.cc:1035] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.579366 9715 consensus_queue.cc:1035] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.581017 9687 mvcc.cc:205] Tried to move back new op lower bound from 7316987513153716224 to 7316987512298823680. Current Snapshot: MvccSnapshot[applied={T|T < 7316987513153716224}]
I20260810 14:58:24.581751 9684 mvcc.cc:205] Tried to move back new op lower bound from 7316987513153716224 to 7316987512298823680. Current Snapshot: MvccSnapshot[applied={T|T < 7316987513153716224}]
I20260810 14:58:24.589042 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.585978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:46735 (local address 127.28.193.193:38239)
0810 14:58:24.586018 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.586020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.586202 (+ 182us) server_negotiation.cc:407] Connection header received
0810 14:58:24.586271 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.586271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.586282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.586328 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.586589 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.587092 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.588305 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.588594 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.588645 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.588851 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.588877 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.588908 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.588964 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.588982 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:58:24.589108 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.585946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38239 (local address 127.28.193.195:46735)
0810 14:58:24.586138 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:24.586150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:24.586213 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:24.586437 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:24.586437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:24.586449 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:58:24.586512 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.586514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.587177 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.587178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.588223 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:24.588227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.588681 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.588682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:24.588753 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.588770 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.588911 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.588913 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:24.588961 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:58:24.589003 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260810 14:58:24.602993 8839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.603128 8839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.603874 8839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Sending DeleteTablet for 3 replicas of tablet 705ce1f1f7694c0b8199939f8a91df43
I20260810 14:58:24.604225 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260810 14:58:24.604223 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 14:58:24.604347 9702 tablet_replica.cc:330] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: stopping tablet replica
I20260810 14:58:24.604415 9702 raft_consensus.cc:2229] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.604393 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet 705ce1f1f7694c0b8199939f8a91df43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41162
I20260810 14:58:24.604451 9702 raft_consensus.cc:2258] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.604473 9699 tablet_replica.cc:330] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: stopping tablet replica
I20260810 14:58:24.604521 9699 raft_consensus.cc:2229] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:24.604517 9701 tablet_replica.cc:330] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: stopping tablet replica
I20260810 14:58:24.604568 9701 raft_consensus.cc:2229] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.604599 9701 raft_consensus.cc:2258] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.604602 9699 raft_consensus.cc:2258] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.604982 9699 ts_tablet_manager.cc:1905] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.604982 9701 ts_tablet_manager.cc:1905] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.606864 9702 ts_tablet_manager.cc:1905] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.606892 9701 ts_tablet_manager.cc:1918] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:24.606941 9701 log.cc:1196] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/705ce1f1f7694c0b8199939f8a91df43
I20260810 14:58:24.606937 9699 ts_tablet_manager.cc:1918] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:24.607000 9699 log.cc:1196] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/705ce1f1f7694c0b8199939f8a91df43
I20260810 14:58:24.607160 9701 ts_tablet_manager.cc:1939] T 705ce1f1f7694c0b8199939f8a91df43 P dba47596ada34b1cacc7837020cca7d2: Deleting consensus metadata
I20260810 14:58:24.607247 9699 ts_tablet_manager.cc:1939] T 705ce1f1f7694c0b8199939f8a91df43 P d937773821df41bcb9f3401ab3174ec9: Deleting consensus metadata
I20260810 14:58:24.607462 8817 catalog_manager.cc:4841] TS dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405): tablet 705ce1f1f7694c0b8199939f8a91df43 (table test-table [id=a237980f7fde460bbd087227038214ae]) successfully deleted
I20260810 14:58:24.607549 8822 catalog_manager.cc:4841] TS d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239): tablet 705ce1f1f7694c0b8199939f8a91df43 (table test-table [id=a237980f7fde460bbd087227038214ae]) successfully deleted
I20260810 14:58:24.608544 9702 ts_tablet_manager.cc:1918] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:24.608598 9702 log.cc:1196] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/705ce1f1f7694c0b8199939f8a91df43
I20260810 14:58:24.608793 9702 ts_tablet_manager.cc:1939] T 705ce1f1f7694c0b8199939f8a91df43 P c30c47cd298d4534bf41ad9b0061c76d: Deleting consensus metadata
I20260810 14:58:24.609344 8822 catalog_manager.cc:4841] TS c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745): tablet 705ce1f1f7694c0b8199939f8a91df43 (table test-table [id=a237980f7fde460bbd087227038214ae]) successfully deleted
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:58:24.641897 8923 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:24.634908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.28.193.254:34703)
0810 14:58:24.634969 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.634971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.635051 (+ 80us) server_negotiation.cc:407] Connection header received
0810 14:58:24.635130 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.635131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.635142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.635219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.635960 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.636749 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.637900 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.638216 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.638306 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.641811 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45850: BlockingRecv error: recv got EOF from 127.0.0.1:45850 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 10 14:58:24 dist-test-slave-cvwc krb5kdc[8763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM
I20260810 14:58:24.652485 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.643336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.28.193.254:34703)
0810 14:58:24.643380 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.643382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.643436 (+ 54us) server_negotiation.cc:407] Connection header received
0810 14:58:24.643513 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.643513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.643523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.643593 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:24.644200 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.645007 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.646083 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.646383 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.646450 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.649286 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:24.649288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:24.649288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:24.649297 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:24.651125 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.651573 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.651574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.651574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.651651 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:24.651883 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:24.651884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:24.651884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:24.652034 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:24.652134 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.652297 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.652385 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 14:58:24.654470 8839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:24.654691 8839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:24.656276 9505 tablet_service.cc:1467] Processing CreateTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 (DEFAULT_TABLE table=test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.656322 9067 tablet_service.cc:1467] Processing CreateTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 (DEFAULT_TABLE table=test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.656443 9505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ba47f8ae4949f181165a8fcc8eb537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.656452 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ba47f8ae4949f181165a8fcc8eb537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.657935 9289 tablet_service.cc:1467] Processing CreateTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 (DEFAULT_TABLE table=test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.657924 9666 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Bootstrap starting.
I20260810 14:58:24.658058 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5ba47f8ae4949f181165a8fcc8eb537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.658496 9666 tablet_bootstrap.cc:654] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.659336 9667 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Bootstrap starting.
I20260810 14:58:24.659746 9667 tablet_bootstrap.cc:654] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.660080 9664 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Bootstrap starting.
I20260810 14:58:24.660392 9664 tablet_bootstrap.cc:654] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.660779 9667 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: No bootstrap required, opened a new log
I20260810 14:58:24.660832 9667 ts_tablet_manager.cc:1397] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:24.660974 9667 raft_consensus.cc:348] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.661043 9667 raft_consensus.cc:374] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.661062 9667 raft_consensus.cc:729] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.661105 9667 consensus_queue.cc:260] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.661197 9667 ts_tablet_manager.cc:1428] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.662178 9664 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: No bootstrap required, opened a new log
I20260810 14:58:24.662235 9664 ts_tablet_manager.cc:1397] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:24.662340 9664 raft_consensus.cc:348] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.662411 9664 raft_consensus.cc:374] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.662431 9664 raft_consensus.cc:729] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.662470 9664 consensus_queue.cc:260] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.662559 9664 ts_tablet_manager.cc:1428] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.666141 9666 tablet_bootstrap.cc:492] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: No bootstrap required, opened a new log
I20260810 14:58:24.666211 9666 ts_tablet_manager.cc:1397] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260810 14:58:24.666386 9666 raft_consensus.cc:348] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.666472 9666 raft_consensus.cc:374] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.666492 9666 raft_consensus.cc:729] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.666534 9666 consensus_queue.cc:260] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.666644 9666 ts_tablet_manager.cc:1428] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:24.670984 9762 raft_consensus.cc:482] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:24.671059 9762 raft_consensus.cc:504] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.671190 9762 leader_election.cc:288] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:24.671391 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ba47f8ae4949f181165a8fcc8eb537" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9" is_pre_election: true
I20260810 14:58:24.671394 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ba47f8ae4949f181165a8fcc8eb537" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d" is_pre_election: true
I20260810 14:58:24.671473 9098 raft_consensus.cc:2454] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:24.671492 9316 raft_consensus.cc:2454] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:24.671619 9439 leader_election.cc:302] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.671738 9762 raft_consensus.cc:2781] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:24.671772 9762 raft_consensus.cc:482] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:24.671789 9762 raft_consensus.cc:3037] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.672685 9762 raft_consensus.cc:504] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.672820 9762 leader_election.cc:288] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Requested vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:24.673003 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ba47f8ae4949f181165a8fcc8eb537" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9"
I20260810 14:58:24.673079 9098 raft_consensus.cc:3037] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.673043 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5ba47f8ae4949f181165a8fcc8eb537" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d"
I20260810 14:58:24.673113 9316 raft_consensus.cc:3037] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:24.673924 9098 raft_consensus.cc:2454] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 1.
I20260810 14:58:24.673969 9316 raft_consensus.cc:2454] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 1.
I20260810 14:58:24.674157 9439 leader_election.cc:302] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:24.674291 9762 raft_consensus.cc:2781] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:24.674348 9762 raft_consensus.cc:686] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 1 LEADER]: Becoming Leader. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Running, Role: LEADER
I20260810 14:58:24.674398 9762 consensus_queue.cc:237] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.674978 8839 catalog_manager.cc:5495] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba47596ada34b1cacc7837020cca7d2 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } health_report { overall_health: HEALTHY } } }
I20260810 14:58:24.681119 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:24.677979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.28.193.195:36405)
0810 14:58:24.678079 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:24.678081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:24.678088 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:24.678112 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:24.678113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:24.678123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:24.678174 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:24.678358 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.679063 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.680249 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:24.680565 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:24.680632 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:24.680808 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:24.680842 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:24.680896 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:24.681021 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 14:58:24.681044 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:58:24.682006 9316 raft_consensus.cc:1261] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.682050 9098 raft_consensus.cc:1261] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:24.682243 9735 consensus_queue.cc:1035] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.682458 9735 consensus_queue.cc:1035] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:24.690227 8839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.690346 8839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:24.691188 8839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Sending DeleteTablet for 3 replicas of tablet f5ba47f8ae4949f181165a8fcc8eb537
I20260810 14:58:24.691452 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 14:58:24.691491 9504 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 14:58:24.691500 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41162
I20260810 14:58:24.691545 9701 tablet_replica.cc:330] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: stopping tablet replica
I20260810 14:58:24.691588 9701 raft_consensus.cc:2229] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:24.691591 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet f5ba47f8ae4949f181165a8fcc8eb537 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:24 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260810 14:58:24.691663 9701 raft_consensus.cc:2258] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.691679 9699 tablet_replica.cc:330] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: stopping tablet replica
I20260810 14:58:24.691728 9699 raft_consensus.cc:2229] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.691771 9699 raft_consensus.cc:2258] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.691838 9702 tablet_replica.cc:330] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: stopping tablet replica
I20260810 14:58:24.691890 9702 raft_consensus.cc:2229] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:24.691926 9702 raft_consensus.cc:2258] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:24.692306 8839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:58:24.692384 9701 ts_tablet_manager.cc:1905] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 14:58:24.692452 8839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:24.693863 9289 tablet_service.cc:1467] Processing CreateTablet for tablet f28631c8a7594addbef08d73b3664928 (DEFAULT_TABLE table=test-table [id=f0974de9190c43c4a6261cf4b968ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.693993 9289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28631c8a7594addbef08d73b3664928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.694080 9701 ts_tablet_manager.cc:1918] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.694124 9701 log.cc:1196] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/f5ba47f8ae4949f181165a8fcc8eb537
I20260810 14:58:24.694335 9701 ts_tablet_manager.cc:1939] T f5ba47f8ae4949f181165a8fcc8eb537 P dba47596ada34b1cacc7837020cca7d2: Deleting consensus metadata
I20260810 14:58:24.695362 9664 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Bootstrap starting.
I20260810 14:58:24.695662 9664 tablet_bootstrap.cc:654] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.695922 9067 tablet_service.cc:1467] Processing CreateTablet for tablet f28631c8a7594addbef08d73b3664928 (DEFAULT_TABLE table=test-table [id=f0974de9190c43c4a6261cf4b968ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.696036 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28631c8a7594addbef08d73b3664928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.696241 9664 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: No bootstrap required, opened a new log
I20260810 14:58:24.696302 9664 ts_tablet_manager.cc:1397] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:24.696410 9664 raft_consensus.cc:348] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.696475 9664 raft_consensus.cc:374] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.696496 9664 raft_consensus.cc:729] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30c47cd298d4534bf41ad9b0061c76d, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.696538 9664 consensus_queue.cc:260] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.696630 9664 ts_tablet_manager.cc:1428] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.697403 9667 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: Bootstrap starting.
I20260810 14:58:24.697901 9667 tablet_bootstrap.cc:654] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.697944 8817 catalog_manager.cc:4841] TS dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405): tablet f5ba47f8ae4949f181165a8fcc8eb537 (table test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]) successfully deleted
W20260810 14:58:24.698068 8817 catalog_manager.cc:4808] TS dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405): delete failed for tablet f5ba47f8ae4949f181165a8fcc8eb537 because the tablet was not found. No further retry: Not found: Tablet not found: f5ba47f8ae4949f181165a8fcc8eb537
I20260810 14:58:24.698242 9504 tablet_service.cc:1467] Processing CreateTablet for tablet f28631c8a7594addbef08d73b3664928 (DEFAULT_TABLE table=test-table [id=f0974de9190c43c4a6261cf4b968ecb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:24.698359 9504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f28631c8a7594addbef08d73b3664928. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:24.699786 9666 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Bootstrap starting.
I20260810 14:58:24.700165 9666 tablet_bootstrap.cc:654] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:24.700730 9666 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: No bootstrap required, opened a new log
I20260810 14:58:24.700774 9666 ts_tablet_manager.cc:1397] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:24.700918 9666 raft_consensus.cc:348] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.700975 9666 raft_consensus.cc:374] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.700990 9666 raft_consensus.cc:729] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.701028 9666 consensus_queue.cc:260] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.701115 9666 ts_tablet_manager.cc:1428] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:24.705101 9702 ts_tablet_manager.cc:1905] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.705116 9699 ts_tablet_manager.cc:1905] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:24.706698 9702 ts_tablet_manager.cc:1918] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.706699 9699 ts_tablet_manager.cc:1918] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:24.706758 9702 log.cc:1196] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/f5ba47f8ae4949f181165a8fcc8eb537
I20260810 14:58:24.706758 9699 log.cc:1196] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/f5ba47f8ae4949f181165a8fcc8eb537
I20260810 14:58:24.706956 9702 ts_tablet_manager.cc:1939] T f5ba47f8ae4949f181165a8fcc8eb537 P c30c47cd298d4534bf41ad9b0061c76d: Deleting consensus metadata
I20260810 14:58:24.706956 9699 ts_tablet_manager.cc:1939] T f5ba47f8ae4949f181165a8fcc8eb537 P d937773821df41bcb9f3401ab3174ec9: Deleting consensus metadata
I20260810 14:58:24.707340 9667 tablet_bootstrap.cc:492] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: No bootstrap required, opened a new log
I20260810 14:58:24.707394 9667 ts_tablet_manager.cc:1397] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260810 14:58:24.707384 8822 catalog_manager.cc:4841] TS c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745): tablet f5ba47f8ae4949f181165a8fcc8eb537 (table test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]) successfully deleted
I20260810 14:58:24.707525 8822 catalog_manager.cc:4841] TS d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239): tablet f5ba47f8ae4949f181165a8fcc8eb537 (table test-table [id=0283eaf137d04b23a92c60f5eb3bfcc9]) successfully deleted
I20260810 14:58:24.707523 9667 raft_consensus.cc:348] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.707587 9667 raft_consensus.cc:374] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:24.707612 9667 raft_consensus.cc:729] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d937773821df41bcb9f3401ab3174ec9, State: Initialized, Role: FOLLOWER
I20260810 14:58:24.707650 9667 consensus_queue.cc:260] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:24.707738 9667 ts_tablet_manager.cc:1428] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:25.027345 9752 raft_consensus.cc:482] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:25.027458 9752 raft_consensus.cc:504] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:25.027616 9752 leader_election.cc:288] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:25.027830 9757 raft_consensus.cc:482] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:25.027915 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d" is_pre_election: true
I20260810 14:58:25.027892 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9" is_pre_election: true
I20260810 14:58:25.028023 9316 raft_consensus.cc:2454] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:25.027930 9757 raft_consensus.cc:504] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:25.028167 9757 leader_election.cc:288] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745), dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405)
I20260810 14:58:25.028239 9098 raft_consensus.cc:2454] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 0.
I20260810 14:58:25.028281 9439 leader_election.cc:302] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2; no voters:
I20260810 14:58:25.028338 9543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2" is_pre_election: true
I20260810 14:58:25.028342 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d" is_pre_election: true
I20260810 14:58:25.028401 9752 raft_consensus.cc:2781] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:25.028417 9316 raft_consensus.cc:2454] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d937773821df41bcb9f3401ab3174ec9 in term 0.
I20260810 14:58:25.028452 9752 raft_consensus.cc:482] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:25.028482 9752 raft_consensus.cc:3037] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:25.028549 8994 leader_election.cc:302] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, d937773821df41bcb9f3401ab3174ec9; no voters:
I20260810 14:58:25.028656 9757 raft_consensus.cc:2781] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:25.028705 9757 raft_consensus.cc:482] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:25.028728 9757 raft_consensus.cc:3037] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:25.029522 9752 raft_consensus.cc:504] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:25.029673 9543 raft_consensus.cc:2379] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d937773821df41bcb9f3401ab3174ec9 in current term 1: Already voted for candidate dba47596ada34b1cacc7837020cca7d2 in this term.
I20260810 14:58:25.029644 9757 raft_consensus.cc:504] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:25.029680 9752 leader_election.cc:288] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Requested vote from peers d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239), c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745)
I20260810 14:58:25.029801 9757 leader_election.cc:288] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 election: Requested vote from peers c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745), dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405)
I20260810 14:58:25.029894 9316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d"
I20260810 14:58:25.029963 9316 raft_consensus.cc:3037] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:25.030042 9098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "dba47596ada34b1cacc7837020cca7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d937773821df41bcb9f3401ab3174ec9"
I20260810 14:58:25.030117 9098 raft_consensus.cc:2379] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dba47596ada34b1cacc7837020cca7d2 in current term 1: Already voted for candidate d937773821df41bcb9f3401ab3174ec9 in this term.
I20260810 14:58:25.030314 9541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba47596ada34b1cacc7837020cca7d2"
I20260810 14:58:25.030412 9541 raft_consensus.cc:2379] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d937773821df41bcb9f3401ab3174ec9 in current term 1: Already voted for candidate dba47596ada34b1cacc7837020cca7d2 in this term.
I20260810 14:58:25.030627 9315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f28631c8a7594addbef08d73b3664928" candidate_uuid: "d937773821df41bcb9f3401ab3174ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30c47cd298d4534bf41ad9b0061c76d"
I20260810 14:58:25.030833 8994 leader_election.cc:302] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d937773821df41bcb9f3401ab3174ec9; no voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2
I20260810 14:58:25.030848 9316 raft_consensus.cc:2454] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba47596ada34b1cacc7837020cca7d2 in term 1.
I20260810 14:58:25.030926 9757 raft_consensus.cc:2726] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 14:58:25.030995 9439 leader_election.cc:302] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c30c47cd298d4534bf41ad9b0061c76d, dba47596ada34b1cacc7837020cca7d2; no voters: d937773821df41bcb9f3401ab3174ec9
I20260810 14:58:25.031098 9752 raft_consensus.cc:2781] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:25.031167 9752 raft_consensus.cc:686] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 LEADER]: Becoming Leader. State: Replica: dba47596ada34b1cacc7837020cca7d2, State: Running, Role: LEADER
I20260810 14:58:25.031226 9752 consensus_queue.cc:237] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } }
I20260810 14:58:25.031898 8843 catalog_manager.cc:5495] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba47596ada34b1cacc7837020cca7d2 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "dba47596ada34b1cacc7837020cca7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba47596ada34b1cacc7837020cca7d2" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 36405 } health_report { overall_health: HEALTHY } } }
I20260810 14:58:25.065560 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:25.063601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56175 (local address 127.28.193.195:36405)
0810 14:58:25.063651 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:25.063653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:25.063659 (+ 6us) server_negotiation.cc:407] Connection header received
0810 14:58:25.063677 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:25.063678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:25.063686 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:25.063726 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:25.063826 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:25.064273 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:25.064951 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:25.065133 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:25.065195 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:25.065358 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:25.065382 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:25.065416 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:25.065461 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 14:58:25.065476 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:58:25.065570 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:25.063418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:36405 (local address 127.28.193.195:56175)
0810 14:58:25.063471 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:25.063485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:25.063535 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:25.063739 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:25.063740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:25.063741 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:58:25.063781 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:25.063782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:25.064327 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:25.064328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:25.064863 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:25.064865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:25.065233 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:25.065234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:25.065287 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:25.065295 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:58:25.065427 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:58:25.065429 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:25.065458 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 14:58:25.065501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 14:58:25.067670 9316 raft_consensus.cc:1261] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:25.067693 9098 raft_consensus.cc:1261] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Refusing update from remote peer dba47596ada34b1cacc7837020cca7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:25.067878 9762 consensus_queue.cc:1035] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d937773821df41bcb9f3401ab3174ec9" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:25.068001 9752 consensus_queue.cc:1035] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30c47cd298d4534bf41ad9b0061c76d" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:25.078116 8843 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:25.078238 8843 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:25.079113 8843 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f127d43dad3449cca285bd8ff3d6de9b: Sending DeleteTablet for 3 replicas of tablet f28631c8a7594addbef08d73b3664928
I20260810 14:58:25.079341 9067 tablet_service.cc:1514] Processing DeleteTablet for tablet f28631c8a7594addbef08d73b3664928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:25 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41162
I20260810 14:58:25.079377 9289 tablet_service.cc:1514] Processing DeleteTablet for tablet f28631c8a7594addbef08d73b3664928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:25 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260810 14:58:25.079385 9505 tablet_service.cc:1514] Processing DeleteTablet for tablet f28631c8a7594addbef08d73b3664928 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:25 UTC) from {username='oryx', principal='oryx/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 14:58:25.079468 9699 tablet_replica.cc:330] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: stopping tablet replica
I20260810 14:58:25.079484 9701 tablet_replica.cc:330] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: stopping tablet replica
I20260810 14:58:25.079517 9699 raft_consensus.cc:2229] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:25.079531 9701 raft_consensus.cc:2229] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:25.079550 9699 raft_consensus.cc:2258] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:25.079583 9701 raft_consensus.cc:2258] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:25.079614 9702 tablet_replica.cc:330] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: stopping tablet replica
I20260810 14:58:25.079663 9702 raft_consensus.cc:2229] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:25.079692 9702 raft_consensus.cc:2258] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:25.079918 9701 ts_tablet_manager.cc:1905] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:25.080024 9699 ts_tablet_manager.cc:1905] T f28631c8a7594addbef08d73b3664928 P d937773821df41bcb9f3401ab3174ec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:25.080117 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 8963
I20260810 14:58:25.083389 9702 ts_tablet_manager.cc:1905] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:25.085080 9702 ts_tablet_manager.cc:1918] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:25.085136 9702 log.cc:1196] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/f28631c8a7594addbef08d73b3664928
I20260810 14:58:25.085376 9702 ts_tablet_manager.cc:1939] T f28631c8a7594addbef08d73b3664928 P c30c47cd298d4534bf41ad9b0061c76d: Deleting consensus metadata
I20260810 14:58:25.089042 8822 catalog_manager.cc:4841] TS c30c47cd298d4534bf41ad9b0061c76d (127.28.193.194:36745): tablet f28631c8a7594addbef08d73b3664928 (table test-table [id=f0974de9190c43c4a6261cf4b968ecb8]) successfully deleted
I20260810 14:58:25.089498 9701 ts_tablet_manager.cc:1918] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:58:25.089555 9701 log.cc:1196] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/f28631c8a7594addbef08d73b3664928
I20260810 14:58:25.089787 9701 ts_tablet_manager.cc:1939] T f28631c8a7594addbef08d73b3664928 P dba47596ada34b1cacc7837020cca7d2: Deleting consensus metadata
I20260810 14:58:25.091967 8817 catalog_manager.cc:4841] TS dba47596ada34b1cacc7837020cca7d2 (127.28.193.195:36405): tablet f28631c8a7594addbef08d73b3664928 (table test-table [id=f0974de9190c43c4a6261cf4b968ecb8]) successfully deleted
W20260810 14:58:25.096916 8822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:38239 (error 108)
I20260810 14:58:25.097520 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:25.097337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38239 (local address 127.0.0.1:41174)
0810 14:58:25.097439 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:25.097485 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
I20260810 14:58:25.097510 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 9184
W20260810 14:58:25.097640 8822 catalog_manager.cc:4584] TS d937773821df41bcb9f3401ab3174ec9 (127.28.193.193:38239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f28631c8a7594addbef08d73b3664928: Network error: Client connection negotiation failed: client connection to 127.28.193.193:38239: connect: Connection refused (error 111)
I20260810 14:58:25.106448 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 9404
I20260810 14:58:25.117365 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 8785
2026-08-10T14:58:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3075 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:25 dist-test-slave-cvwc krb5kdc[9828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:25 dist-test-slave-cvwc krb5kdc[9828](info): set up 2 sockets
Aug 10 14:58:25 dist-test-slave-cvwc krb5kdc[9828](info): commencing operation
krb5kdc: starting...
W20260810 14:58:26.295601 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:26 dist-test-slave-cvwc krb5kdc[9828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:26.414837 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33501
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:33501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:26.496052 9847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:26.496253 9847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:26.496272 9847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:26.497385 9847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:26.497424 9847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:26.497440 9847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:26.497462 9847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:26.497478 9847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:26.499063 9847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:33501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:26.499281 9847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:26.499497 9847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:26.504096 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:26.504143 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:26.504320 9847 server_base.cc:1034] running on GCE node
W20260810 14:58:26.504094 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:26.504642 9847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:26.504926 9847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:26.506105 9847 hybrid_clock.cc:648] HybridClock initialized: now 1786373906506089 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T14:58:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1386 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:26 dist-test-slave-cvwc krb5kdc[9870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:26 dist-test-slave-cvwc krb5kdc[9870](info): set up 2 sockets
Aug 10 14:58:26 dist-test-slave-cvwc krb5kdc[9870](info): commencing operation
krb5kdc: starting...
W20260810 14:58:27.678514 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:27 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:27.804113 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41669
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:27.886691 9886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:27.886907 9886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:27.886929 9886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:27.888044 9886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:27.888080 9886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:27.888093 9886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:27.888104 9886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:27.888115 9886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:27.889535 9886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:27.889770 9886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:27.889982 9886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:27.894919 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:27.894928 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:27.894958 9886 server_base.cc:1034] running on GCE node
W20260810 14:58:27.894928 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:27.895362 9886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:27.895627 9886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:27.896797 9886 hybrid_clock.cc:648] HybridClock initialized: now 1786373907896778 us; error 29 us; skew 500 ppm
Aug 10 14:58:27 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:27.899441 9886 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:27.899927 9886 webserver.cc:460] Webserver started at http://127.28.193.254:35247/ using document root <none> and password file <none>
I20260810 14:58:27.900138 9886 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:27.900187 9886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:27.900331 9886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:27.901405 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "0c33e1877fec4bb18e9e9533a0521a87"
format_stamp: "Formatted at 2026-08-10 14:58:27 on dist-test-slave-cvwc"
server_key: "5d9195aed28d899243395c9b7b94689d"
server_key_iv: "980bb5ead9c47cb31227e6311c09e92e"
server_key_version: "encryptionkey@0"
I20260810 14:58:27.901801 9886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "0c33e1877fec4bb18e9e9533a0521a87"
format_stamp: "Formatted at 2026-08-10 14:58:27 on dist-test-slave-cvwc"
server_key: "5d9195aed28d899243395c9b7b94689d"
server_key_iv: "980bb5ead9c47cb31227e6311c09e92e"
server_key_version: "encryptionkey@0"
I20260810 14:58:27.903146 9886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:27.903929 9910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:27.904165 9886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:27.904265 9886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "0c33e1877fec4bb18e9e9533a0521a87"
format_stamp: "Formatted at 2026-08-10 14:58:27 on dist-test-slave-cvwc"
server_key: "5d9195aed28d899243395c9b7b94689d"
server_key_iv: "980bb5ead9c47cb31227e6311c09e92e"
server_key_version: "encryptionkey@0"
I20260810 14:58:27.904335 9886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:27.917050 9886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:27.919615 9886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:27.919760 9886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:27.923805 9886 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:41669
I20260810 14:58:27.923859 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:41669 every 8 connection(s)
I20260810 14:58:27.924185 9886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:27.925072 9991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:27.927376 9991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Bootstrap starting.
I20260810 14:58:27.927908 9991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:27.928146 9991 log.cc:824] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:27.928900 9991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: No bootstrap required, opened a new log
I20260810 14:58:27.929414 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 9886
I20260810 14:58:27.929533 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:27.929752 29447 external_mini_cluster.cc:1428] Setting key 77bbbf84f8a7a3b8691376b151be42b7
I20260810 14:58:27.931084 9991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:27.931236 9991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:27.931262 9991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c33e1877fec4bb18e9e9533a0521a87, State: Initialized, Role: FOLLOWER
I20260810 14:58:27.931378 9991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:27.931458 9991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:27.931489 9991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:27.931581 9991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:27.932318 9991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:27.932417 9991 leader_election.cc:302] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c33e1877fec4bb18e9e9533a0521a87; no voters:
I20260810 14:58:27.932587 9991 leader_election.cc:288] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:27.932614 9998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:27.932726 9998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 LEADER]: Becoming Leader. State: Replica: 0c33e1877fec4bb18e9e9533a0521a87, State: Running, Role: LEADER
I20260810 14:58:27.932866 9991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:27.932893 9998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:27.933283 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c33e1877fec4bb18e9e9533a0521a87. Latest consensus state: current_term: 1 leader_uuid: "0c33e1877fec4bb18e9e9533a0521a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } } }
I20260810 14:58:27.933295 10000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c33e1877fec4bb18e9e9533a0521a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } } }
I20260810 14:58:27.933476 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:27.933485 10000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:27.933755 10015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:27.934350 10015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:27.936034 10015 catalog_manager.cc:1347] Generated new cluster ID: 466293a131db446ca848c8fb86bcf3f4
I20260810 14:58:27.936089 10015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:27 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:27.942348 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:27.930316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.28.193.254:41669)
0810 14:58:27.930483 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:27.930492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:27.930517 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:27.931198 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:27.931206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:27.931280 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:27.931430 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:27.932240 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:27.933561 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:27.934665 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:27.935150 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:27.935267 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:27.937894 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:27.937898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:27.937898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:27.937910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:27.940445 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:27.940811 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:27.940814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:27.940814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:27.940912 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:27.941068 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:27.941068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:27.941069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:27.941343 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:27.941476 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:27.942005 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 14:58:27.942136 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260810 14:58:27.945415 10015 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:27.945945 10015 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:27.952816 10015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Generated new TSK 0
I20260810 14:58:27.953002 10015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:27.985628 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:28.067936 10039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.068156 10039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.068188 10039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.069365 10039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.069408 10039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.069444 10039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:28.070897 10039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.071141 10039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.071359 10039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.076251 10057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.076283 10049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.076308 10052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.076452 10039 server_base.cc:1034] running on GCE node
I20260810 14:58:28.076709 10039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.076948 10039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.078110 10039 hybrid_clock.cc:648] HybridClock initialized: now 1786373908078095 us; error 29 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.080690 10039 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:28.081166 10039 webserver.cc:460] Webserver started at http://127.28.193.193:41783/ using document root <none> and password file <none>
I20260810 14:58:28.081364 10039 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.081405 10039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.081501 10039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:28.082566 10039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "6afde271402348488518655178c1a317"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "3ffcb111aa7745bb5c9b0150de426794"
server_key_iv: "8da6e4bea9274e7b6f940c695c82939b"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.082876 10039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "6afde271402348488518655178c1a317"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "3ffcb111aa7745bb5c9b0150de426794"
server_key_iv: "8da6e4bea9274e7b6f940c695c82939b"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.084076 10039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 14:58:28.084713 10068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.084877 10039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:28.084962 10039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "6afde271402348488518655178c1a317"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "3ffcb111aa7745bb5c9b0150de426794"
server_key_iv: "8da6e4bea9274e7b6f940c695c82939b"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.085026 10039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.101483 10039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.103605 10039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.103762 10039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.104005 10039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:28.104318 10039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:28.104353 10039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.104382 10039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:28.104398 10039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.110277 10039 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:46287
I20260810 14:58:28.110380 10252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:46287 every 8 connection(s)
I20260810 14:58:28.110657 10039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:28.111352 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 10039
I20260810 14:58:28.111476 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:28.111673 29447 external_mini_cluster.cc:1428] Setting key 15d69b3b805d6f9176b12b7af4684dbe
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:28.121078 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.111463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:36617 (local address 127.28.193.254:41669)
0810 14:58:28.111529 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.111532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.111783 (+ 251us) server_negotiation.cc:407] Connection header received
0810 14:58:28.112338 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.112339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.112353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.112412 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.113348 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.114083 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.115414 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.115604 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.115650 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.118024 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:28.118026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:28.118026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:28.118035 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:28.119522 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.120016 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.120018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.120018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.120082 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.120323 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.120324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.120325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.120469 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:28.120555 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:28.120883 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 14:58:28.120977 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:58:28.121130 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.111412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:36617)
0810 14:58:28.111649 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.111679 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.112217 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.112471 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.112475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.113040 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.113226 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.113232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.114183 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.114185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.115341 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.115344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.115678 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.115679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.115857 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.116330 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.116341 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.117929 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:28.119690 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.119698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.119700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.119931 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.120153 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.120153 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.120154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.120254 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.120598 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:28.120606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:28.120729 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 14:58:28.120829 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:58:28.120957 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 14:58:28.121690 10255 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41669
I20260810 14:58:28.121827 10255 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:28.121986 10255 heartbeater.cc:511] Master 127.28.193.254:41669 requested a full tablet report, sending...
I20260810 14:58:28.122584 9940 ts_manager.cc:194] Registered new tserver with Master: 6afde271402348488518655178c1a317 (127.28.193.193:46287)
I20260810 14:58:28.123193 9940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:36617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:28.140769 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:28.223116 10260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.223315 10260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.223336 10260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.224540 10260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.224589 10260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.224615 10260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:28.226065 10260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.226281 10260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.226481 10260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.231325 10268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.231460 10273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.231469 10260 server_base.cc:1034] running on GCE node
W20260810 14:58:28.231406 10270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.231779 10260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.232012 10260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.233145 10260 hybrid_clock.cc:648] HybridClock initialized: now 1786373908233125 us; error 40 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.235885 10260 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:28.236446 10260 webserver.cc:460] Webserver started at http://127.28.193.194:35745/ using document root <none> and password file <none>
I20260810 14:58:28.236708 10260 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.236776 10260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.236922 10260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:28.238389 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "bc24e47271e748d48bf40ae48b177388"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "bf9754c7352c98e66c43bb5daf6a68b1"
server_key_iv: "ca4e493451d92b1e77b0a44f828735fc"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.238857 10260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "bc24e47271e748d48bf40ae48b177388"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "bf9754c7352c98e66c43bb5daf6a68b1"
server_key_iv: "ca4e493451d92b1e77b0a44f828735fc"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.240622 10260 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:58:28.241536 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.241694 10260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:28.241789 10260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "bc24e47271e748d48bf40ae48b177388"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "bf9754c7352c98e66c43bb5daf6a68b1"
server_key_iv: "ca4e493451d92b1e77b0a44f828735fc"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.241844 10260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.266654 10260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.269037 10260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.269160 10260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.269424 10260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:28.269794 10260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:28.269836 10260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.269860 10260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:28.269870 10260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.276575 10260 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:34231
I20260810 14:58:28.276648 10484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:34231 every 8 connection(s)
I20260810 14:58:28.276971 10260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:28.277354 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 10260
I20260810 14:58:28.277459 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:28.277635 29447 external_mini_cluster.cc:1428] Setting key 95bd7eed1f06b2cc466991778540429b
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 14:58:28.287853 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.277788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:55327 (local address 127.28.193.254:41669)
0810 14:58:28.277866 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.277868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.278107 (+ 239us) server_negotiation.cc:407] Connection header received
0810 14:58:28.278947 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.278948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.278960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.279016 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.280306 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.281067 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.282198 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.282472 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.282553 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.284960 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:28.284962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:28.284962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:28.284971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:28.286523 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.286924 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.286926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.286926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.286988 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.287222 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.287223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.287223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.287366 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:28.287436 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:28.287719 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:58:28.287784 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:58:28.288067 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.277757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:55327)
0810 14:58:28.277948 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.277997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.278589 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.279124 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.279130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.279884 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.280161 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.280168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.281189 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.281190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.282108 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.282111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.282591 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.282592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.282712 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.283300 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.283308 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.284854 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:28.286609 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.286615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.286617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.286843 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.287049 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.287050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.287050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.287147 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.287430 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:28.287435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:28.287598 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 14:58:28.287699 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 14:58:28.287847 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260810 14:58:28.288527 10485 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41669
I20260810 14:58:28.288618 10485 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:28.288815 10485 heartbeater.cc:511] Master 127.28.193.254:41669 requested a full tablet report, sending...
I20260810 14:58:28.289234 9940 ts_manager.cc:194] Registered new tserver with Master: bc24e47271e748d48bf40ae48b177388 (127.28.193.194:34231)
I20260810 14:58:28.289621 9940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:55327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:28.305239 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:28.388983 10493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.389192 10493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.389225 10493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.390424 10493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.390470 10493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.390503 10493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:28.391930 10493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.392166 10493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.392385 10493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.397289 10502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.397315 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.397352 10493 server_base.cc:1034] running on GCE node
W20260810 14:58:28.397315 10509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.397670 10493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.397955 10493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.399094 10493 hybrid_clock.cc:648] HybridClock initialized: now 1786373908399080 us; error 29 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.401566 10493 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:28.402045 10493 webserver.cc:460] Webserver started at http://127.28.193.195:46017/ using document root <none> and password file <none>
I20260810 14:58:28.402252 10493 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.402302 10493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.402432 10493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:28.403496 10493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "8de45a8b85f34ced9dd342feebc0420c"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "5ca09d6b82007928f4e5a9c6ca95f60f"
server_key_iv: "3f4c6adb027f6f36b98db95374f98480"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.403846 10493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "8de45a8b85f34ced9dd342feebc0420c"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "5ca09d6b82007928f4e5a9c6ca95f60f"
server_key_iv: "3f4c6adb027f6f36b98db95374f98480"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.405123 10493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:28.405881 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.406049 10493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:28.406133 10493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "8de45a8b85f34ced9dd342feebc0420c"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "5ca09d6b82007928f4e5a9c6ca95f60f"
server_key_iv: "3f4c6adb027f6f36b98db95374f98480"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.406200 10493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.417896 10493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.420027 10493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.420181 10493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.420433 10493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:28.420785 10493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:28.420833 10493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.420866 10493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:28.420882 10493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.426627 10493 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:38849
I20260810 14:58:28.426690 10711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:38849 every 8 connection(s)
I20260810 14:58:28.427018 10493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:28.431092 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 10493
I20260810 14:58:28.431203 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:28.431401 29447 external_mini_cluster.cc:1428] Setting key 768ab741a82a5302decf83ece0bfdc25
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:28.436426 9997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.427820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:35573 (local address 127.28.193.254:41669)
0810 14:58:28.427911 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.427914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.428134 (+ 220us) server_negotiation.cc:407] Connection header received
0810 14:58:28.428626 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.428628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.428639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.428692 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.429647 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.430215 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.431214 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.431508 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.431579 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.434057 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:28.434058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:28.434058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:28.434065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:28.435353 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.435709 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.435710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.435710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.435756 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:28.435924 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:28.435924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:28.435924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:28.436021 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:28.436065 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:28.436305 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 14:58:28.436362 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:58:28.436638 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.427781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:35573)
0810 14:58:28.427991 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.428041 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.428532 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.428772 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.428776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.429353 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.429553 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.429558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.430303 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.430304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.431106 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.431108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.431607 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.431608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.431754 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.432292 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.432303 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.433974 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:28.435426 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.435432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.435434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.435649 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.435797 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:28.435797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:28.435798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:28.435883 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:28.436080 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:28.436084 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:28.436200 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 14:58:28.436280 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:58:28.436400 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
I20260810 14:58:28.437151 10712 heartbeater.cc:347] Connected to a master server at 127.28.193.254:41669
I20260810 14:58:28.437244 10712 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:28.437404 10712 heartbeater.cc:511] Master 127.28.193.254:41669 requested a full tablet report, sending...
I20260810 14:58:28.437770 9940 ts_manager.cc:194] Registered new tserver with Master: 8de45a8b85f34ced9dd342feebc0420c (127.28.193.195:38849)
I20260810 14:58:28.438156 9940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:35573
I20260810 14:58:28.443596 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
Entry for principal oryx/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab.
WARNING: no policy specified for oryx/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
Entry for principal oryx/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab.
WARNING: no policy specified for oryx/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
Entry for principal oryx/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab.
I20260810 14:58:28.514209 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 10039
I20260810 14:58:28.526023 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 10260
I20260810 14:58:28.538034 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 10493
I20260810 14:58:28.550006 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 9886
I20260810 14:58:28.563982 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41669
--webserver_interface=127.28.193.254
--webserver_port=35247
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:28.647689 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.648074 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.648156 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.649452 10722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:28.649514 10722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.649536 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.649555 10722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:28.649574 10722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:28.651268 10722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=35247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.651566 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.651820 10722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.656924 10730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.656929 10733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.657012 10722 server_base.cc:1034] running on GCE node
W20260810 14:58:28.656971 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.657405 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.657756 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.658936 10722 hybrid_clock.cc:648] HybridClock initialized: now 1786373908658922 us; error 29 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.664531 10722 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:28.665047 10722 webserver.cc:460] Webserver started at http://127.28.193.254:35247/ using document root <none> and password file <none>
I20260810 14:58:28.665277 10722 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.665349 10722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.666862 10722 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:28.667718 10747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.667975 10722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:28.668071 10722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "0c33e1877fec4bb18e9e9533a0521a87"
format_stamp: "Formatted at 2026-08-10 14:58:27 on dist-test-slave-cvwc"
server_key: "5d9195aed28d899243395c9b7b94689d"
server_key_iv: "980bb5ead9c47cb31227e6311c09e92e"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.668448 10722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.677254 10722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.679287 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.679440 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.683490 10722 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:41669
I20260810 14:58:28.683532 10831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:41669 every 8 connection(s)
I20260810 14:58:28.683923 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:28.685408 10835 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:58:28.686151 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Bootstrap starting.
I20260810 14:58:28.689448 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 10722
I20260810 14:58:28.689836 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:46287
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:41669
--webserver_port=41783
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:58:28.694845 10835 log.cc:824] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:28.696768 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:58:28.697064 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Bootstrap complete.
I20260810 14:58:28.699234 10835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:28.699465 10835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c33e1877fec4bb18e9e9533a0521a87, State: Initialized, Role: FOLLOWER
I20260810 14:58:28.699608 10835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:28.699678 10835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:28.699723 10835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:28.699774 10835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:58:28.700775 10835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:28.700879 10835 leader_election.cc:302] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c33e1877fec4bb18e9e9533a0521a87; no voters:
I20260810 14:58:28.701058 10835 leader_election.cc:288] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:58:28.701110 10841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:58:28.701269 10841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [term 2 LEADER]: Becoming Leader. State: Replica: 0c33e1877fec4bb18e9e9533a0521a87, State: Running, Role: LEADER
I20260810 14:58:28.701344 10835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:28.701431 10841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } }
I20260810 14:58:28.701771 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c33e1877fec4bb18e9e9533a0521a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } } }
I20260810 14:58:28.701853 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:28.702183 10850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:28.702224 10843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c33e1877fec4bb18e9e9533a0521a87. Latest consensus state: current_term: 2 leader_uuid: "0c33e1877fec4bb18e9e9533a0521a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c33e1877fec4bb18e9e9533a0521a87" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 41669 } } }
I20260810 14:58:28.702291 10843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:28.702694 10850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:28.702833 10850 catalog_manager.cc:1259] Loaded cluster ID: 466293a131db446ca848c8fb86bcf3f4
I20260810 14:58:28.703065 10850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:28.704257 10850 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:28.704434 10850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c33e1877fec4bb18e9e9533a0521a87: Loaded TSK: 0
I20260810 14:58:28.704640 10850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:58:28.774302 10838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.774518 10838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.774541 10838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.775753 10838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.775799 10838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.775825 10838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:28.777284 10838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.193.keytab
--principal=oryx/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:46287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=41783
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.777496 10838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.777750 10838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.782910 10881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.782930 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.782912 10879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.783169 10838 server_base.cc:1034] running on GCE node
I20260810 14:58:28.783380 10838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.783617 10838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.784799 10838 hybrid_clock.cc:648] HybridClock initialized: now 1786373908784777 us; error 40 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.787513 10838 init.cc:377] Logged in from keytab as oryx/127.28.193.193@KRBTEST.COM (short username oryx)
I20260810 14:58:28.787964 10838 webserver.cc:460] Webserver started at http://127.28.193.193:41783/ using document root <none> and password file <none>
I20260810 14:58:28.788170 10838 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.788236 10838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.789618 10838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:28.790270 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.790437 10838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 14:58:28.790534 10838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "6afde271402348488518655178c1a317"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "3ffcb111aa7745bb5c9b0150de426794"
server_key_iv: "8da6e4bea9274e7b6f940c695c82939b"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.790982 10838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.806577 10838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.808967 10838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.809129 10838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.809405 10838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:28.809837 10838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:28.809891 10838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.809922 10838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:28.809950 10838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.817283 10838 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:46287
I20260810 14:58:28.817353 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:46287 every 8 connection(s)
I20260810 14:58:28.817793 10838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.826172 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 10838
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.826491 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:34231
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:41669
--webserver_port=35745
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:58:28.827317 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.818598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:56395)
0810 14:58:28.818799 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.818846 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.819453 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.819863 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.819867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.820575 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.820839 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.820846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.821993 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.821994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.823204 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.823208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.823836 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.823837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.823972 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.824565 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.824574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.827121 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
W20260810 14:58:28.827476 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.818726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:56395 (local address 127.28.193.254:41669)
0810 14:58:28.818920 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.818927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.818955 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:58:28.819638 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.819643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.819685 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.819794 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.820947 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.821897 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.823318 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.823742 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.823839 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.827293 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:56395: BlockingRecv error: recv got EOF from 127.28.193.193:56395 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
W20260810 14:58:28.827795 11092 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:41669 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.193.254:41669: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.832727 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.828037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:52495)
0810 14:58:28.828231 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.828241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.828282 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.828457 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.828458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.828574 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.828616 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.828617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.829238 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.829239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.829773 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.829775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.830170 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.830171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.830244 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.830651 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.830656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.832659 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175}
W20260810 14:58:28.832828 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.828135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:52495 (local address 127.28.193.254:41669)
0810 14:58:28.828344 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.828346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.828353 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:28.828373 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.828373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.828382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.828426 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.828690 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.829160 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.829844 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.830093 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.830158 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.832768 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:52495: BlockingRecv error: recv got EOF from 127.28.193.193:52495 (error 108)
Metrics: {"server-negotiator.queue_time_us":190}
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.838415 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.833239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:40423)
0810 14:58:28.833314 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.833326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.833371 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.833507 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.833508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.833644 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.833678 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.833680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.834544 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.834545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.835289 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.835292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.835717 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.835719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.835774 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.836291 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.836296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.838346 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 14:58:28.838577 11092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:58:28.838518 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.833312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:40423 (local address 127.28.193.254:41669)
0810 14:58:28.833348 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.833350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.833358 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:28.833422 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.833423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.833431 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.833481 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.833783 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.834450 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.835384 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.835633 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.835702 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.838446 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:40423: BlockingRecv error: recv got EOF from 127.28.193.193:40423 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 14:58:28.913175 11100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:28.913390 11100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:28.913422 11100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:28.914603 11100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:28.914649 11100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:28.914683 11100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:28.916199 11100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.194.keytab
--principal=oryx/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:34231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=35745
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:28.916462 11100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:28.916682 11100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:28.922106 11110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.922298 11107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:28.922115 11106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:28.922559 11100 server_base.cc:1034] running on GCE node
I20260810 14:58:28.922847 11100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:28.923128 11100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:28.924304 11100 hybrid_clock.cc:648] HybridClock initialized: now 1786373908924292 us; error 27 us; skew 500 ppm
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373908, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:28.926971 11100 init.cc:377] Logged in from keytab as oryx/127.28.193.194@KRBTEST.COM (short username oryx)
I20260810 14:58:28.927397 11100 webserver.cc:460] Webserver started at http://127.28.193.194:35745/ using document root <none> and password file <none>
I20260810 14:58:28.927634 11100 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:28.927698 11100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:28.929087 11100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:28.929656 11127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.929831 11100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:28.929919 11100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "bc24e47271e748d48bf40ae48b177388"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "bf9754c7352c98e66c43bb5daf6a68b1"
server_key_iv: "ca4e493451d92b1e77b0a44f828735fc"
server_key_version: "encryptionkey@0"
I20260810 14:58:28.930289 11100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:28.943413 11100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:28.945574 11100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:28.945775 11100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:28.946018 11100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:28.946326 11100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:28.946359 11100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.946389 11100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:28.946406 11100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:28.951828 11100 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:34231
I20260810 14:58:28.951891 11315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:34231 every 8 connection(s)
I20260810 14:58:28.952224 11100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:28.953451 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 11100
I20260810 14:58:28.953764 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:38849
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:41669
--webserver_port=46017
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.959997 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.952958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:40813)
0810 14:58:28.953101 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.953138 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.953650 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.953922 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.953926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.954483 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.954677 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.954681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.955483 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.955485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.956292 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.956295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.956699 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.956700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.956825 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.957371 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.957380 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.959844 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
W20260810 14:58:28.960040 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.952974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:40813 (local address 127.28.193.254:41669)
0810 14:58:28.953038 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.953040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.953241 (+ 201us) server_negotiation.cc:407] Connection header received
0810 14:58:28.953780 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.953781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.953798 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.953842 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.954779 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.955368 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.956372 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.956615 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.956685 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.959975 (+ 3290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:40813: BlockingRecv error: recv got EOF from 127.28.193.194:40813 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 14:58:28.960210 11316 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:41669 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.193.254:41669: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.965073 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.960460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:52937)
0810 14:58:28.960566 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.960575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.960615 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.960739 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.960740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.960848 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.960891 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.960892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.961760 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.961761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.962267 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.962269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.962734 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.962734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.962784 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.963213 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.963216 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.965023 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260810 14:58:28.965209 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.960523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:52937 (local address 127.28.193.254:41669)
0810 14:58:28.960593 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.960595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.960606 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:28.960663 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.960664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.960674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.960731 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.960958 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.961639 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.962386 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.962663 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.962736 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.965147 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:52937: BlockingRecv error: recv got EOF from 127.28.193.194:52937 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:28 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:28.969811 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:28.965505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:33543)
0810 14:58:28.965606 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:28.965616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:28.965662 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:28.965804 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:28.965805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:28.965911 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:28.965937 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.965938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.966806 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.966806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.967275 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:28.967276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.967700 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.967701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:28.967733 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.968135 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:28.968139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:28.969746 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 14:58:28.969933 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:28.965564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:33543 (local address 127.28.193.254:41669)
0810 14:58:28.965624 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:28.965626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:28.965651 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:28.965701 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:28.965702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:28.965734 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:28.965787 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:28.965999 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.966727 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.967348 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:28.967641 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:28.967701 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:28.969875 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:33543: BlockingRecv error: recv got EOF from 127.28.193.194:33543 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 14:58:28.969936 11316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:58:29.039450 11323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:29.039655 11323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:29.039676 11323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:29.040810 11323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:29.040850 11323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:29.040874 11323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:29.042323 11323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/krb5kdc/oryx_127.28.193.195.keytab
--principal=oryx/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:38849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=46017
--tserver_master_addrs=127.28.193.254:41669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:29.042536 11323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:29.042737 11323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:29.047966 11323 server_base.cc:1034] running on GCE node
W20260810 14:58:29.047869 11337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:29.047914 11334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:29.047869 11332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:29.048323 11323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:29.048580 11323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:29.049829 11323 hybrid_clock.cc:648] HybridClock initialized: now 1786373909049808 us; error 28 us; skew 500 ppm
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373909, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:29.052354 11323 init.cc:377] Logged in from keytab as oryx/127.28.193.195@KRBTEST.COM (short username oryx)
I20260810 14:58:29.052807 11323 webserver.cc:460] Webserver started at http://127.28.193.195:46017/ using document root <none> and password file <none>
I20260810 14:58:29.053021 11323 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:29.053085 11323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:29.054587 11323 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:29.055271 11353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:29.055505 11323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:29.055600 11323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "8de45a8b85f34ced9dd342feebc0420c"
format_stamp: "Formatted at 2026-08-10 14:58:28 on dist-test-slave-cvwc"
server_key: "5ca09d6b82007928f4e5a9c6ca95f60f"
server_key_iv: "3f4c6adb027f6f36b98db95374f98480"
server_key_version: "encryptionkey@0"
I20260810 14:58:29.055979 11323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:29.084589 11323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:29.086714 11323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:29.086875 11323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:29.087121 11323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:29.087440 11323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:29.087476 11323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:29.087507 11323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:29.087522 11323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:29.093050 11323 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:38849
I20260810 14:58:29.093116 11525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:38849 every 8 connection(s)
I20260810 14:58:29.093452 11323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.100063 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 11323
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.101138 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.094211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:51197)
0810 14:58:29.094427 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:29.094475 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:29.094998 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:29.095294 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:29.095298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:29.095936 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:29.096136 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.096141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.096849 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.096850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.097655 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.097657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.098038 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.098039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.098162 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.098687 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:29.098695 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:29.100939 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
W20260810 14:58:29.101145 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:29.094294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:51197 (local address 127.28.193.254:41669)
0810 14:58:29.094405 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.094408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.094567 (+ 159us) server_negotiation.cc:407] Connection header received
0810 14:58:29.095111 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.095113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.095125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.095189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.096244 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.096747 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.097759 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.097960 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.098018 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.101075 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:51197: BlockingRecv error: recv got EOF from 127.28.193.195:51197 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 14:58:29.101373 11526 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:41669 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.193.254:41669: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.106259 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.101621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:33149)
0810 14:58:29.101679 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:29.101687 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:29.101737 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:29.101869 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:29.101870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:29.102008 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:29.102057 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.102059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.102884 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.102884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.103367 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.103368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.103747 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.103748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.103808 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.104201 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:29.104205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:29.106197 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:58:29.106429 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:29.101689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:33149 (local address 127.28.193.254:41669)
0810 14:58:29.101762 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.101764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.101770 (+ 6us) server_negotiation.cc:407] Connection header received
0810 14:58:29.101798 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.101799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.101807 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.101851 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.102146 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.102799 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.103454 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.103667 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.103748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.106356 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:33149: BlockingRecv error: recv got EOF from 127.28.193.195:33149 (error 108)
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":54}
I20260810 14:58:29.109114 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.100600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.28.193.254:41669)
0810 14:58:29.100737 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.100739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.100748 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:29.100774 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.100775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.100810 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.100889 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.101983 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.102621 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.103447 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.103641 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.103690 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.105183 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:29.105188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:29.105189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:29.105202 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:29.107707 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:29.108055 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:29.108059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:29.108060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:29.108163 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:29.108355 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:29.108356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:29.108356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:29.108609 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:29.108740 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:29.108896 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 14:58:29.109021 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.110782 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.106677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:36849)
0810 14:58:29.106727 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:29.106740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:29.106784 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:29.106918 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:29.106918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:29.107025 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:29.107055 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.107056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.107853 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.107854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.108342 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.108344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.108683 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.108684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.108736 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.109074 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:29.109079 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:29.110738 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 14:58:29.110885 11098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:29.106752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:36849 (local address 127.28.193.254:41669)
0810 14:58:29.106794 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.106795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.106806 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:29.106829 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.106830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.106839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.106907 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.107126 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.107793 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.108431 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.108616 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.108667 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.110813 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:36849: BlockingRecv error: recv got EOF from 127.28.193.195:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 14:58:29.110936 11526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.843500 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.838983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:47825)
0810 14:58:29.839112 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:29.839123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:29.839176 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:29.839312 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:29.839313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:29.839486 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:29.839539 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.839540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.840191 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.840192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.840844 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.840848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.841212 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.841213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.841286 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.841729 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:29.841734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:29.843430 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
W20260810 14:58:29.843597 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:29.839071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:47825 (local address 127.28.193.254:41669)
0810 14:58:29.839195 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.839197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.839205 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:29.839228 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.839229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.839239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.839301 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.839597 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.840090 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.840925 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.841135 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.841193 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.843526 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:47825: BlockingRecv error: recv got EOF from 127.28.193.193:47825 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:29 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:29.975210 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:29.970486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:54575)
0810 14:58:29.970661 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:29.970673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:29.970748 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:29.970887 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:29.970887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:29.971076 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:29.971128 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.971130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.971823 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.971824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.972433 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:29.972436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.972891 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.972892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:29.972938 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.973350 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:29.973354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:29.975121 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
W20260810 14:58:29.975286 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:29.970602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:54575 (local address 127.28.193.254:41669)
0810 14:58:29.970661 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:29.970663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:29.970754 (+ 91us) server_negotiation.cc:407] Connection header received
0810 14:58:29.970800 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:29.970801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:29.970814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:29.970870 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:29.971226 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.971739 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.972520 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:29.972823 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:29.972877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:29.975233 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:54575: BlockingRecv error: recv got EOF from 127.28.193.194:54575 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:30.115985 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:30.111396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:60059)
0810 14:58:30.111529 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:30.111541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:30.111598 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:30.111725 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:30.111726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:30.111915 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:30.111967 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.111969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.112622 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.112623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.113173 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.113175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.113617 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.113618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.113683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.114211 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:30.114216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:30.115912 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
W20260810 14:58:30.116070 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:30.111530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:60059 (local address 127.28.193.254:41669)
0810 14:58:30.111590 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:30.111592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:30.111603 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:30.111639 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:30.111640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:30.111654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:30.111712 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:30.112056 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.112559 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.113255 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.113522 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.113593 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.116009 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:60059: BlockingRecv error: recv got EOF from 127.28.193.195:60059 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:30.849187 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:30.843960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:54813)
0810 14:58:30.844094 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:30.844103 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:30.844155 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:30.844289 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:30.844289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:30.844459 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:30.844510 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.844511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.845162 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.845163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.846056 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.846060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.846484 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.846485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.846549 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.847069 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:30.847075 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:30.849104 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260810 14:58:30.849300 11539 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:30.844037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54813 (local address 127.28.193.254:41669)
0810 14:58:30.844167 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:30.844169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:30.844194 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:30.844215 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:30.844216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:30.844227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:30.844276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:30.844568 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.845063 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.846139 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.846396 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.846464 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.849230 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:54813: BlockingRecv error: recv got EOF from 127.28.193.193:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:30 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:30.980031 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:30.975729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:57287)
0810 14:58:30.975865 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:30.975877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:30.975935 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:30.976061 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:30.976062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:30.976227 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:30.976278 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.976280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.976930 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.976930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.977444 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:30.977453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.977775 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.977775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:30.977815 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.978254 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:30.978258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:30.979971 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
W20260810 14:58:30.980129 11539 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:30.975849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:57287 (local address 127.28.193.254:41669)
0810 14:58:30.975939 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:30.975940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:30.975952 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:58:30.975977 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:30.975978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:30.975990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:30.976047 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:30.976374 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.976865 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.977511 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:30.977693 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:30.977772 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:30.980067 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:57287: BlockingRecv error: recv got EOF from 127.28.193.194:57287 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:31.120811 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:31.116515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:37429)
0810 14:58:31.116660 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:31.116675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:31.116749 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:31.116874 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:31.116875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:31.117054 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:31.117107 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.117109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.117750 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.117751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.118280 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.118282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.118596 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.118596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.118632 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.119050 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:31.119054 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:31.120752 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
W20260810 14:58:31.120906 11539 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:31.116692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:37429 (local address 127.28.193.254:41669)
0810 14:58:31.116763 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:31.116765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:31.116772 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:31.116797 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:31.116798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:31.116808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:31.116855 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:31.117172 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.117665 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.118344 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.118539 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.118588 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.120846 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:37429: BlockingRecv error: recv got EOF from 127.28.193.195:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:31.854882 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:31.850282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:39695)
0810 14:58:31.850407 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:31.850418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:31.850494 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:31.850678 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:31.850679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:31.850857 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:31.850921 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.850923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.851676 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.851677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.852213 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.852215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.852531 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.852532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.852570 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.852996 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:31.853000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:31.854821 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260810 14:58:31.855008 11546 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:31.850387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:39695 (local address 127.28.193.254:41669)
0810 14:58:31.850528 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:31.850530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:31.850541 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:31.850569 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:31.850570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:31.850583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:31.850645 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:31.851021 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.851605 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.852283 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.852479 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.852524 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.854929 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:39695: BlockingRecv error: recv got EOF from 127.28.193.193:39695 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":55,"threads_started":1}
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:31 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:31.984874 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:31.980553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:38507)
0810 14:58:31.980709 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:31.980721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:31.980795 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:31.980948 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:31.980949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:31.981163 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:31.981215 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.981217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.981869 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.981869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.982390 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:31.982392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.982699 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.982699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:31.982750 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.983204 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:31.983209 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:31.984814 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
W20260810 14:58:31.984972 11546 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:31.980619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:38507 (local address 127.28.193.254:41669)
0810 14:58:31.980667 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:31.980669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:31.980790 (+ 121us) server_negotiation.cc:407] Connection header received
0810 14:58:31.980854 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:31.980855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:31.980869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:31.980924 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:31.981288 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.981805 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.982454 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:31.982626 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:31.982669 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:31.984912 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:38507: BlockingRecv error: recv got EOF from 127.28.193.194:38507 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:32.125958 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:32.121366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:35837)
0810 14:58:32.121532 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:32.121543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:32.121619 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:32.121836 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:32.121837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:32.122011 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:32.122070 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.122072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.122754 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.122755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.123277 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.123279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.123584 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.123584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.123622 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.124044 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:32.124048 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:32.125889 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
W20260810 14:58:32.126072 11546 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:32.121435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:35837 (local address 127.28.193.254:41669)
0810 14:58:32.121485 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:32.121488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:32.121639 (+ 151us) server_negotiation.cc:407] Connection header received
0810 14:58:32.121670 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:32.121671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:32.121691 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:32.121786 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:32.122141 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.122683 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.123343 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.123525 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.123576 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.126002 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:35837: BlockingRecv error: recv got EOF from 127.28.193.195:35837 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:32.860036 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:32.855366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:38145)
0810 14:58:32.855541 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:32.855558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:32.855630 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:32.855758 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:32.855759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:32.855926 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:32.855976 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.855978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.856622 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.856623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.857162 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.857165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.857535 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.857536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.857579 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.858067 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:32.858073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:32.859977 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
W20260810 14:58:32.860133 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:32.855461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38145 (local address 127.28.193.254:41669)
0810 14:58:32.855569 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:32.855571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:32.855626 (+ 55us) server_negotiation.cc:407] Connection header received
0810 14:58:32.855688 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:32.855688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:32.855699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:32.855745 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:32.856034 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.856537 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.857235 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.857474 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.857526 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.860065 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:38145: BlockingRecv error: recv got EOF from 127.28.193.193:38145 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:32 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:32.989609 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:32.985357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:39249)
0810 14:58:32.985510 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:32.985522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:32.985592 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:32.985737 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:32.985738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:32.985916 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:32.985974 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.985976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.986577 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.986578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.987100 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:32.987102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.987390 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.987391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:32.987433 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.987834 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:32.987838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:32.989549 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
W20260810 14:58:32.989706 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:32.985419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:39249 (local address 127.28.193.254:41669)
0810 14:58:32.985473 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:32.985475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:32.985585 (+ 110us) server_negotiation.cc:407] Connection header received
0810 14:58:32.985639 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:32.985640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:32.985650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:32.985700 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:32.986031 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.986519 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.987157 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:32.987335 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:32.987380 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:32.989640 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:39249: BlockingRecv error: recv got EOF from 127.28.193.194:39249 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:33.131042 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:33.126458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:37451)
0810 14:58:33.126610 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:33.126621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:33.126679 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:33.126821 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:33.126821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:33.126999 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:33.127049 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.127051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.127666 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.127666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.128185 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.128186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.128588 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.128589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.128642 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.129142 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:33.129147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:33.130976 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
W20260810 14:58:33.131111 11553 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:33.126531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:37451 (local address 127.28.193.254:41669)
0810 14:58:33.126580 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:33.126582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:33.126675 (+ 93us) server_negotiation.cc:407] Connection header received
0810 14:58:33.126748 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:33.126749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:33.126763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:33.126804 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:33.127101 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.127606 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.128261 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.128506 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.128575 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.131058 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:37451: BlockingRecv error: recv got EOF from 127.28.193.195:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:33.869549 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:33.860494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:38707)
0810 14:58:33.860637 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:33.860647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:33.860698 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:33.860902 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:33.860903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:33.861070 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:33.861121 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.861123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.861867 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.861868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.862398 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.862399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.862787 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.862788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.862833 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.867610 (+ 4777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:33.867616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:33.869464 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260810 14:58:33.869653 11559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:33.860612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38707 (local address 127.28.193.254:41669)
0810 14:58:33.860766 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:33.860768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:33.860777 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:33.860805 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:33.860806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:33.860818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:33.860893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:33.861205 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.861790 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.862489 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.862691 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.862781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.869588 (+ 6807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:38707: BlockingRecv error: recv got EOF from 127.28.193.193:38707 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:33 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:33.994606 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:33.990118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:53051)
0810 14:58:33.990252 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:33.990264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:33.990318 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:33.990466 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:33.990466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:33.990641 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:33.990690 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.990692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.991302 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.991303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.991828 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:33.991830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.992118 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.992119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:33.992157 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.992600 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:33.992605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:33.994523 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
W20260810 14:58:33.994753 11559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:33.990282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:53051 (local address 127.28.193.254:41669)
0810 14:58:33.990340 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:33.990342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:33.990352 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:33.990382 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:33.990383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:33.990394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:33.990443 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:33.990745 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.991238 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.991889 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:33.992068 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:33.992111 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:33.994639 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:53051: BlockingRecv error: recv got EOF from 127.28.193.194:53051 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:34.136278 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:34.131659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:56937)
0810 14:58:34.131849 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:34.131864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:34.131949 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:34.132089 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:34.132090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:34.132262 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:34.132311 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.132314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.132947 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.132948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.133498 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.133500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.133884 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.133885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.133926 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.134397 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:34.134402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:34.136219 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
W20260810 14:58:34.136365 11559 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:34.131763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56937 (local address 127.28.193.254:41669)
0810 14:58:34.131840 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:34.131842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:34.131924 (+ 82us) server_negotiation.cc:407] Connection header received
0810 14:58:34.132007 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:34.132008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:34.132024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:34.132074 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:34.132369 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.132885 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.133598 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.133826 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.133882 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.136308 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:56937: BlockingRecv error: recv got EOF from 127.28.193.195:56937 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:34.874709 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:34.870040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:43245)
0810 14:58:34.870182 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:34.870197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:34.870279 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:34.870411 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:34.870412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:34.870579 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:34.870628 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.870630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.871307 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.871308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.871840 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.871842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.872213 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.872214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.872289 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.872770 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:34.872774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:34.874648 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260810 14:58:34.874831 11568 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:34.870115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:43245 (local address 127.28.193.254:41669)
0810 14:58:34.870271 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:34.870273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:34.870283 (+ 10us) server_negotiation.cc:407] Connection header received
0810 14:58:34.870329 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:34.870329 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:34.870340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:34.870399 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:34.870678 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.871224 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.871930 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.872138 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.872193 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.874739 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:43245: BlockingRecv error: recv got EOF from 127.28.193.193:43245 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:34 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:34.999866 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:34.995072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:57709)
0810 14:58:34.995221 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:34.995232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:34.995305 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:34.995451 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:34.995452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:34.995632 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:34.995682 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.995685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.996435 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.996436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.997210 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:34.997213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.997545 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.997546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:34.997602 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.998094 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:34.998098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:34.999798 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
W20260810 14:58:34.999953 11568 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:34.995164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:57709 (local address 127.28.193.254:41669)
0810 14:58:34.995219 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:34.995222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:34.995294 (+ 72us) server_negotiation.cc:407] Connection header received
0810 14:58:34.995360 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:34.995361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:34.995374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:34.995441 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:34.995776 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.996346 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.997291 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:34.997482 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:34.997539 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:34.999894 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:57709: BlockingRecv error: recv got EOF from 127.28.193.194:57709 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:35.140954 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:35.136730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:34207)
0810 14:58:35.136879 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:35.136889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:35.136947 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:35.137054 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:35.137055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:35.137217 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:35.137265 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:35.137267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.137885 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.137886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:35.138406 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:35.138408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.138700 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.138701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:35.138736 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:35.139188 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:35.139192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:35.140894 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
W20260810 14:58:35.141050 11568 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:35.136803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:34207 (local address 127.28.193.254:41669)
0810 14:58:35.136863 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:35.136865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:35.136930 (+ 65us) server_negotiation.cc:407] Connection header received
0810 14:58:35.136987 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:35.136988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:35.137000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:35.137049 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:35.137321 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.137830 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.138462 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.138652 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.138696 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:35.140987 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:34207: BlockingRecv error: recv got EOF from 127.28.193.195:34207 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:35.880180 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:35.875224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:33197)
0810 14:58:35.875369 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:35.875380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:35.875434 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:35.875597 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:35.875598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:35.875770 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:35.875822 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:35.875824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.876545 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.876547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:35.877091 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:35.877093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.877493 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.877495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:35.877552 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:35.878024 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:35.878029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:35.880121 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260810 14:58:35.880303 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:35.875333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33197 (local address 127.28.193.254:41669)
0810 14:58:35.875474 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:35.875476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:35.875484 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:35.875509 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:35.875510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:35.875520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:35.875586 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:35.875938 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.876457 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.877180 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:35.877389 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:35.877454 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:35.880228 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:33197: BlockingRecv error: recv got EOF from 127.28.193.193:33197 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:35 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:36.004842 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:36.000304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:39737)
0810 14:58:36.000425 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:36.000436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:36.000492 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:36.000607 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:36.000608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:36.000803 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:36.000856 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.000858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.001560 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.001561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.002206 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.002208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.002531 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.002531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.002587 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.003050 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:36.003056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:36.004782 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
W20260810 14:58:36.004987 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:36.000380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:39737 (local address 127.28.193.254:41669)
0810 14:58:36.000459 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:36.000461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:36.000473 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:58:36.000545 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:36.000545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:36.000556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:36.000599 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:36.000945 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.001443 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.002275 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.002469 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.002523 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.004909 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:39737: BlockingRecv error: recv got EOF from 127.28.193.194:39737 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:58:36 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:36 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:36.145879 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:36.141395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:46037)
0810 14:58:36.141559 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:36.141573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:36.141643 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:36.141808 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:36.141809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:36.141989 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:36.142039 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.142041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.142688 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.142689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.143209 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.143211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.143550 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.143551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.143607 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.144076 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:36.144081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:36.145800 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
W20260810 14:58:36.145954 11577 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:36.141484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:46037 (local address 127.28.193.254:41669)
0810 14:58:36.141567 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:36.141570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:36.141622 (+ 52us) server_negotiation.cc:407] Connection header received
0810 14:58:36.141707 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:36.141708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:36.141744 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:36.141792 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:36.142118 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.142608 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.143292 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.143483 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.143538 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.145897 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:46037: BlockingRecv error: recv got EOF from 127.28.193.195:46037 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 14:58:36 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:36 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:36.885020 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:36.880670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:50491)
0810 14:58:36.880798 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:36.880809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:36.880857 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:36.881016 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:36.881016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:36.881178 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:36.881229 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.881231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.881929 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.881929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.882447 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:36.882448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.882757 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.882757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:36.882792 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.883192 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:36.883196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:36.884926 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260810 14:58:36.885085 11586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:36.880786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:50491 (local address 127.28.193.254:41669)
0810 14:58:36.880898 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:36.880899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:36.880907 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:36.880935 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:36.880936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:36.880945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:36.881007 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:36.881321 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.881868 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.882514 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:36.882701 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:36.882752 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:36.885026 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:50491: BlockingRecv error: recv got EOF from 127.28.193.193:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:37.010167 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:37.005347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:51631)
0810 14:58:37.005488 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:37.005499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:37.005564 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:37.005684 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:37.005685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:37.005887 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:37.005938 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.005940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.006762 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.006763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.007294 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.007295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.007598 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.007598 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.007637 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.008156 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:37.008161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:37.010095 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260810 14:58:37.010248 11586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:37.005397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:51631 (local address 127.28.193.254:41669)
0810 14:58:37.005481 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:37.005484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:37.005559 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:58:37.005620 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:37.005621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:37.005632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:37.005672 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:37.006040 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.006697 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.007359 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.007540 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.007588 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.010189 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:51631: BlockingRecv error: recv got EOF from 127.28.193.194:51631 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:37.150713 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:37.146443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:51309)
0810 14:58:37.146600 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:37.146610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:37.146662 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:37.146819 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:37.146819 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:37.146982 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:37.147031 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.147033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.147672 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.147673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.148180 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.148181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.148463 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.148464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.148501 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.148915 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:37.148919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:37.150634 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
W20260810 14:58:37.150787 11586 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:37.146551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:51309 (local address 127.28.193.254:41669)
0810 14:58:37.146635 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:37.146637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:37.146649 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:58:37.146719 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:37.146719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:37.146732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:37.146805 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:37.147115 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.147611 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.148236 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.148407 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.148449 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.150735 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:51309: BlockingRecv error: recv got EOF from 127.28.193.195:51309 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:37 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:37.890393 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:37.885544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:37843)
0810 14:58:37.885695 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:37.885706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:37.885832 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:37.886020 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:37.886021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:37.886196 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:37.886249 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.886251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.887046 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.887047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.887635 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:37.887638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.887988 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.887990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:37.888038 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.888500 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:37.888504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:37.890329 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
W20260810 14:58:37.890491 11593 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:37.885644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:37843 (local address 127.28.193.254:41669)
0810 14:58:37.885776 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:37.885779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:37.885809 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:58:37.885919 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:37.885919 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:37.885932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:37.886004 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:37.886307 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.886933 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.887714 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:37.887915 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:37.887969 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:37.890421 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:37843: BlockingRecv error: recv got EOF from 127.28.193.193:37843 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":51,"threads_started":1}
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:38.014952 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:38.010653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:50297)
0810 14:58:38.010800 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:38.010810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:38.010867 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:38.010989 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:38.010989 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:38.011159 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:38.011209 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.011211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.011848 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.011849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.012365 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.012366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.012665 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.012666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.012704 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.013173 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:38.013178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:38.014892 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
W20260810 14:58:38.015056 11593 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:38.010752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:50297 (local address 127.28.193.254:41669)
0810 14:58:38.010807 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:38.010810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:38.010859 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:58:38.010912 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:38.010913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:38.010924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:38.010975 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:38.011270 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.011777 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.012430 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.012614 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.012656 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.014989 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:50297: BlockingRecv error: recv got EOF from 127.28.193.194:50297 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:38.155694 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:38.151361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:49433)
0810 14:58:38.151504 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:38.151516 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:38.151577 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:38.151700 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:38.151701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:38.151871 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:38.151919 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.151921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.152562 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.152563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.153091 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.153093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.153413 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.153414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.153460 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.153901 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:38.153905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:38.155623 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
W20260810 14:58:38.155782 11593 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:38.151433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:49433 (local address 127.28.193.254:41669)
0810 14:58:38.151508 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:38.151510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:38.151569 (+ 59us) server_negotiation.cc:407] Connection header received
0810 14:58:38.151625 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:38.151626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:38.151639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:38.151687 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:38.151989 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.152492 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.153161 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.153346 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.153392 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.155715 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:49433: BlockingRecv error: recv got EOF from 127.28.193.195:49433 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:38 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:38.905018 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:38.890843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:42439)
0810 14:58:38.891004 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:38.891018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:38.891086 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:38.891224 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:38.891225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:38.891387 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:38.891439 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.891440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.892047 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.892047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.892553 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:38.892554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.892828 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.892828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:38.892863 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.903034 (+ 10171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:38.903040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:38.904937 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20260810 14:58:38.905122 11598 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:38.890901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:42439 (local address 127.28.193.254:41669)
0810 14:58:38.891041 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:38.891042 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:38.891066 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:58:38.891146 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:38.891147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:38.891160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:38.891214 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:38.891490 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.891992 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.892604 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:38.892781 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:38.892824 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:38.905048 (+ 12224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:42439: BlockingRecv error: recv got EOF from 127.28.193.193:42439 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:39.020138 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:39.015428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:59999)
0810 14:58:39.015575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:39.015588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:39.015661 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:39.015790 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:39.015791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:39.015994 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:39.016042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.016044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.016726 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.016727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.017261 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.017263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.017598 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.017599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.017653 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.018096 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:39.018101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:39.020046 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260810 14:58:39.020207 11598 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:39.015515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:59999 (local address 127.28.193.254:41669)
0810 14:58:39.015579 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:39.015581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:39.015639 (+ 58us) server_negotiation.cc:407] Connection header received
0810 14:58:39.015716 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:39.015717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:39.015730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:39.015779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:39.016133 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.016631 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.017340 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.017537 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.017592 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.020152 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:59999: BlockingRecv error: recv got EOF from 127.28.193.194:59999 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:39.160781 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:39.156255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:39955)
0810 14:58:39.156403 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:39.156417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:39.156482 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:39.156610 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:39.156611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:39.156778 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:39.156834 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.156836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.157481 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.157481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.158095 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.158098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.158446 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.158446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.158495 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.158932 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:39.158936 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:39.160722 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260810 14:58:39.160892 11598 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:39.156377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:39955 (local address 127.28.193.254:41669)
0810 14:58:39.156439 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:39.156441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:39.156466 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:39.156541 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:39.156541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:39.156556 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:39.156599 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:39.156908 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.157418 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.158180 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.158374 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.158453 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.160831 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:39955: BlockingRecv error: recv got EOF from 127.28.193.195:39955 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:39 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:39.909708 11602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:39.905475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:58481)
0810 14:58:39.905606 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:39.905617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:39.905670 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:39.905821 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:39.905822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:39.905998 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:39.906048 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.906050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.906655 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.906656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.907212 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:39.907214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.907499 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.907499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:39.907534 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.907935 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:39.907939 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:39.909635 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
W20260810 14:58:39.909808 11603 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:39.905545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:58481 (local address 127.28.193.254:41669)
0810 14:58:39.905648 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:39.905650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:39.905659 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:39.905728 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:39.905729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:39.905742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:39.905802 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:39.906099 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.906594 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.907267 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:39.907452 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:39.907496 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:39.909746 (+ 2250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:58481: BlockingRecv error: recv got EOF from 127.28.193.193:58481 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:40.025022 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:40.020622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:54545)
0810 14:58:40.020754 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:40.020765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:40.020829 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:40.020943 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:40.020944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:40.021116 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:40.021163 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.021165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.021799 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.021799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.022432 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.022435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.022773 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.022774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.022825 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.023231 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:40.023234 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:40.024950 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
W20260810 14:58:40.025116 11603 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:40.020706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:54545 (local address 127.28.193.254:41669)
0810 14:58:40.020756 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:40.020758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:40.020812 (+ 54us) server_negotiation.cc:407] Connection header received
0810 14:58:40.020882 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:40.020882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:40.020893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:40.020929 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:40.021221 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.021735 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.022507 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.022705 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.022757 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.025049 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:54545: BlockingRecv error: recv got EOF from 127.28.193.194:54545 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:40.165931 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:40.161280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:44585)
0810 14:58:40.161417 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:40.161429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:40.161493 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:40.161623 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:40.161624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:40.161816 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:40.161865 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.161866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.162550 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.162551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.163106 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.163107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.163396 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.163397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.163432 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.163919 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:40.163923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:40.165831 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260810 14:58:40.166064 11603 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:40.161360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44585 (local address 127.28.193.254:41669)
0810 14:58:40.161439 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:40.161441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:40.161488 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:58:40.161539 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:40.161539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:40.161552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:40.161610 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:40.161930 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.162474 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.163163 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.163344 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.163399 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.165935 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:44585: BlockingRecv error: recv got EOF from 127.28.193.195:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:40 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:40.914626 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:40.910294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:44633)
0810 14:58:40.910425 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:40.910437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:40.910504 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:40.910647 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:40.910647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:40.910817 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:40.910863 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.910864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.911474 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.911475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.911989 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:40.911991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.912315 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.912316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:40.912378 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.912825 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:40.912829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:40.914565 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260810 14:58:40.914736 11611 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:40.910400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:44633 (local address 127.28.193.254:41669)
0810 14:58:40.910534 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:40.910536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:40.910544 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:40.910566 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:40.910567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:40.910577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:40.910636 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:40.910915 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.911421 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.912045 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:40.912234 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:40.912292 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:40.914658 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:44633: BlockingRecv error: recv got EOF from 127.28.193.193:44633 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:41.029979 11612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:41.025479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:59057)
0810 14:58:41.025614 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:41.025629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:41.025702 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:41.025859 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:41.025860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:41.026043 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:41.026088 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.026090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.026748 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.026749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.027267 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.027269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.027599 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.027600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.027657 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.028091 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:41.028095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:41.029907 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260810 14:58:41.030097 11611 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:41.025554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:59057 (local address 127.28.193.254:41669)
0810 14:58:41.025632 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:41.025634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:41.025692 (+ 58us) server_negotiation.cc:407] Connection header received
0810 14:58:41.025784 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:41.025785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:41.025798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:41.025843 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:41.026160 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.026669 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.027344 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.027538 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.027593 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.030030 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:59057: BlockingRecv error: recv got EOF from 127.28.193.194:59057 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:41.171130 11613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:41.166535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:33775)
0810 14:58:41.166675 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:41.166685 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:41.166746 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:41.166865 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:41.166866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:41.167038 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:41.167087 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.167089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.167695 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.167695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.168225 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.168227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.168507 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.168507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.168545 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.168994 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:41.168999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:41.171044 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
W20260810 14:58:41.171190 11611 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:41.166600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:33775 (local address 127.28.193.254:41669)
0810 14:58:41.166650 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:41.166653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:41.166739 (+ 86us) server_negotiation.cc:407] Connection header received
0810 14:58:41.166790 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:41.166791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:41.166804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:41.166852 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:41.167135 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.167637 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.168280 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.168460 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.168503 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.171121 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:33775: BlockingRecv error: recv got EOF from 127.28.193.195:33775 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:41 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:41.919808 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:41.915122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:44309)
0810 14:58:41.915257 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:41.915271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:41.915362 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:41.915509 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:41.915509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:41.915688 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:41.915738 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.915740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.916431 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.916431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.916973 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:41.916975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.917308 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.917309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:41.917346 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.917933 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:41.917938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:41.919734 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
W20260810 14:58:41.919940 11618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:41.915245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:44309 (local address 127.28.193.254:41669)
0810 14:58:41.915386 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:41.915388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:41.915397 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:41.915422 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:41.915423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:41.915434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:41.915492 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:41.915847 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.916363 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.917048 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:41.917239 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:41.917301 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:41.919854 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:44309: BlockingRecv error: recv got EOF from 127.28.193.193:44309 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:42.034828 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:42.030450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:35717)
0810 14:58:42.030614 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:42.030624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:42.030672 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:42.030812 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:42.030812 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:42.030987 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:42.031036 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.031038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.031694 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.031695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.032220 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.032221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.032507 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.032507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.032544 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.032981 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:42.032986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:42.034756 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
W20260810 14:58:42.034951 11618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:42.030591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:35717 (local address 127.28.193.254:41669)
0810 14:58:42.030658 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:42.030660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:42.030671 (+ 11us) server_negotiation.cc:407] Connection header received
0810 14:58:42.030741 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:42.030742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:42.030753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:42.030800 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:42.031116 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.031610 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.032280 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.032459 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.032505 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.034883 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:35717: BlockingRecv error: recv got EOF from 127.28.193.194:35717 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:42.176331 11627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:42.171631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:57909)
0810 14:58:42.171800 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:42.171811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:42.171889 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:42.172014 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:42.172015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:42.172201 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:42.172250 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.172252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.172884 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.172885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.173415 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.173417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.173775 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.173776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.173835 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.174358 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:42.174363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:42.176259 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
W20260810 14:58:42.176416 11618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:42.171692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:57909 (local address 127.28.193.254:41669)
0810 14:58:42.171748 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:42.171750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:42.171870 (+ 120us) server_negotiation.cc:407] Connection header received
0810 14:58:42.171940 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:42.171941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:42.171956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:42.172002 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:42.172321 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.172812 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.173491 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.173680 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.173761 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.176356 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:57909: BlockingRecv error: recv got EOF from 127.28.193.195:57909 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:42 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:42.925081 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:42.920335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:35735)
0810 14:58:42.920453 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:42.920464 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:42.920520 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:42.920659 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:42.920660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:42.920837 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:42.920887 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.920888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.921750 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.921751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.922288 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:42.922290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.922647 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.922648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:42.922703 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.923145 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:42.923150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:42.924990 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260810 14:58:42.925163 11629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:42.920437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:35735 (local address 127.28.193.254:41669)
0810 14:58:42.920546 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:42.920548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:42.920556 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:58:42.920579 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:42.920579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:42.920589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:42.920644 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:42.920977 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.921657 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.922371 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:42.922581 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:42.922636 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:42.925098 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:35735: BlockingRecv error: recv got EOF from 127.28.193.193:35735 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:43.039963 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:43.035268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:37199)
0810 14:58:43.035391 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:43.035402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:43.035462 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:43.035594 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:43.035595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:43.035781 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:43.035829 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.035831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.036458 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.036459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.036987 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.036988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.037408 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.037409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.037470 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.037994 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:43.037999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:43.039901 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260810 14:58:43.040062 11629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:43.035330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:37199 (local address 127.28.193.254:41669)
0810 14:58:43.035387 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:43.035390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:43.035455 (+ 65us) server_negotiation.cc:407] Connection header received
0810 14:58:43.035516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:43.035517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:43.035528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:43.035576 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:43.035879 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.036384 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.037083 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.037310 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.037384 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.040001 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:37199: BlockingRecv error: recv got EOF from 127.28.193.194:37199 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.195@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:43.181339 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:43.176843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.195:44325)
0810 14:58:43.176982 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:43.176996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:43.177066 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:43.177195 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:43.177196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:43.177386 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:43.177439 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.177440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.178138 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.178139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.178683 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.178685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.179005 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.179006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.179062 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.179553 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:43.179559 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:43.181262 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
W20260810 14:58:43.181425 11629 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:43.176959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44325 (local address 127.28.193.254:41669)
0810 14:58:43.177010 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:43.177012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:43.177046 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:58:43.177119 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:43.177119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:43.177134 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:43.177179 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:43.177515 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.178053 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.178758 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.178942 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.178999 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.181374 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.195:44325: BlockingRecv error: recv got EOF from 127.28.193.195:44325 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:43 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.193@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:43.934197 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:43.925701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.193:33445)
0810 14:58:43.925866 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:43.925898 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:43.925965 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:43.926112 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:43.926113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:43.926284 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:43.926335 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.926337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.926942 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.926943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.927464 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:43.927466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.927750 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.927750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:43.927785 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.932292 (+ 4507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:43.932297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:43.934123 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
W20260810 14:58:43.934276 11633 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:43.925777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33445 (local address 127.28.193.254:41669)
0810 14:58:43.925872 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:43.925874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:43.925976 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:58:43.926024 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:43.926024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:43.926039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:43.926106 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:43.926393 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.926887 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.927522 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:43.927702 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:43.927746 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:43.934212 (+ 6466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.193:33445: BlockingRecv error: recv got EOF from 127.28.193.193:33445 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":49,"threads_started":1}
Aug 10 14:58:44 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:58:44 dist-test-slave-cvwc krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.193.194@KRBTEST.COM for oryx/127.28.193.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:58:44.045090 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.040505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:41669 (local address 127.28.193.194:57227)
0810 14:58:44.040641 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:44.040651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:44.040710 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:44.040871 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:44.040872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:44.041034 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:44.041089 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.041091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.041762 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.041764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.042323 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.042325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.042673 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.042674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.042711 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.043208 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:44.043213 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:44.044998 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.193.254:41669: Server oryx/127.28.193.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
W20260810 14:58:44.045154 11633 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:44.040586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:57227 (local address 127.28.193.254:41669)
0810 14:58:44.040637 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.040639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.040696 (+ 57us) server_negotiation.cc:407] Connection header received
0810 14:58:44.040792 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.040793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.040803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.040857 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.041140 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.041632 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.042416 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.042614 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.042671 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.045102 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.193.194:57227: BlockingRecv error: recv got EOF from 127.28.193.194:57227 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 14:58:44.108198 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 10838
I20260810 14:58:44.119799 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 11100
I20260810 14:58:44.127827 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 11323
I20260810 14:58:44.134155 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 10722
2026-08-10T14:58:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17627 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T14:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:44Z Disabled control of system clock
I20260810 14:58:44.167622 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36151
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:58:44.248219 11640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:44.248418 11640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:44.248436 11640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:44.249557 11640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:44.249589 11640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:44.249600 11640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:44.249611 11640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:44.249621 11640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:44.250975 11640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:36151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:44.251181 11640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:44.251461 11640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:58:44.256253 11640 server_base.cc:1034] running on GCE node
W20260810 14:58:44.256208 11647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.256300 11646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.256220 11649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.256626 11640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:44.256922 11640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:44.258059 11640 hybrid_clock.cc:648] HybridClock initialized: now 1786373924258048 us; error 27 us; skew 500 ppm
I20260810 14:58:44.259266 11640 webserver.cc:460] Webserver started at http://127.28.193.254:45375/ using document root <none> and password file <none>
I20260810 14:58:44.259493 11640 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:44.259544 11640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:44.259675 11640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:44.260717 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "2fa3e0537b824487821a3d5bb80c4d0b"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "9df1cc5d854808fb7be0be2cecaf5489"
server_key_iv: "d019a3e0dd78e50ab38330164e1fd72f"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.261065 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "2fa3e0537b824487821a3d5bb80c4d0b"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "9df1cc5d854808fb7be0be2cecaf5489"
server_key_iv: "d019a3e0dd78e50ab38330164e1fd72f"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.262475 11640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:44.263201 11663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.263360 11640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:44.263444 11640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "2fa3e0537b824487821a3d5bb80c4d0b"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "9df1cc5d854808fb7be0be2cecaf5489"
server_key_iv: "d019a3e0dd78e50ab38330164e1fd72f"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.263511 11640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:44.297132 11640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:44.297477 11640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:44.297616 11640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:44.301519 11640 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:36151
I20260810 14:58:44.301575 11740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:36151 every 8 connection(s)
I20260810 14:58:44.301956 11640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:44.302660 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:44.305217 11741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b: Bootstrap starting.
I20260810 14:58:44.305794 11741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:44.306049 11741 log.cc:824] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:44.306658 11741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b: No bootstrap required, opened a new log
I20260810 14:58:44.308145 11741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } }
I20260810 14:58:44.308280 11741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:44.308300 11741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa3e0537b824487821a3d5bb80c4d0b, State: Initialized, Role: FOLLOWER
I20260810 14:58:44.308414 11741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } }
I20260810 14:58:44.308490 11741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:44.308516 11741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:44.308547 11741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:44.309011 11741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } }
I20260810 14:58:44.309094 11741 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa3e0537b824487821a3d5bb80c4d0b; no voters:
I20260810 14:58:44.309209 11741 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:44.309273 11748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:44.309424 11741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:44.309439 11748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [term 1 LEADER]: Becoming Leader. State: Replica: 2fa3e0537b824487821a3d5bb80c4d0b, State: Running, Role: LEADER
I20260810 14:58:44.309617 11748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } }
I20260810 14:58:44.310024 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } } }
I20260810 14:58:44.310139 11752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa3e0537b824487821a3d5bb80c4d0b. Latest consensus state: current_term: 1 leader_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa3e0537b824487821a3d5bb80c4d0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36151 } } }
I20260810 14:58:44.310240 11752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:44.310176 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:44.310753 11764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:44.311254 11764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:44.312150 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 11640
I20260810 14:58:44.312234 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:44.312405 29447 external_mini_cluster.cc:1428] Setting key b7dbe677af6222d151ca9406c6857ea3
I20260810 14:58:44.312925 11764 catalog_manager.cc:1347] Generated new cluster ID: 674c9a949da642e6980209ec906dde23
I20260810 14:58:44.312980 11764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:44.317219 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.312977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44660 (local address 127.28.193.254:36151)
0810 14:58:44.313162 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.313169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.313192 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:58:44.313284 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.313290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.313349 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.313472 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.313665 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.314637 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.315391 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.315838 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.315952 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.316579 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:44.316583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:44.316584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:44.316753 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:44.316786 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:44.316797 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:44.316852 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:44.316930 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:58:44.317009 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260810 14:58:44.324517 11764 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:44.325037 11764 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:44.331991 11764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa3e0537b824487821a3d5bb80c4d0b: Generated new TSK 0
I20260810 14:58:44.332149 11764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:44.333407 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--builtin_ntp_servers=127.28.193.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 14:58:44.423995 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:44.424209 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:44.424233 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:44.425379 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:44.425422 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:44.425446 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:44.426831 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:44.427057 11789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:44.427357 11789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:44.432143 11798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.432133 11799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.432133 11801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.432718 11789 server_base.cc:1034] running on GCE node
I20260810 14:58:44.432917 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:44.433153 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:44.434307 11789 hybrid_clock.cc:648] HybridClock initialized: now 1786373924434287 us; error 30 us; skew 500 ppm
I20260810 14:58:44.435705 11789 webserver.cc:460] Webserver started at http://127.28.193.193:39155/ using document root <none> and password file <none>
I20260810 14:58:44.435954 11789 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:44.436010 11789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:44.436139 11789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:44.437506 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "a12bbd9e4fb04a19824b39c75ae1bfc3"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "73f355148e32de80583c250646fea6d1"
server_key_iv: "7015c265adfab29b3061adc1b6ae3389"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.437924 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "a12bbd9e4fb04a19824b39c75ae1bfc3"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "73f355148e32de80583c250646fea6d1"
server_key_iv: "7015c265adfab29b3061adc1b6ae3389"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.439628 11789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 14:58:44.440475 11810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.440629 11789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:44.440723 11789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "a12bbd9e4fb04a19824b39c75ae1bfc3"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "73f355148e32de80583c250646fea6d1"
server_key_iv: "7015c265adfab29b3061adc1b6ae3389"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.440794 11789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:44.456856 11789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:44.457120 11789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:44.457216 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:44.457404 11789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:44.457683 11789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:44.457741 11789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.457777 11789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:44.457799 11789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.463348 11789 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:33925
I20260810 14:58:44.463403 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:33925 every 8 connection(s)
I20260810 14:58:44.463747 11789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:44.468804 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.464555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:59601 (local address 127.28.193.254:36151)
0810 14:58:44.464624 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.464627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.464844 (+ 217us) server_negotiation.cc:407] Connection header received
0810 14:58:44.465344 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.465345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.465355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.465411 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.466117 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.466626 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.467585 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.467814 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.467877 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.468566 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:44.468567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:44.468567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:44.468572 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:44.468589 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:44.468593 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:44.468638 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:44.468697 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 14:58:44.468742 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:58:44.468937 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.464548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36151 (local address 127.28.193.193:59601)
0810 14:58:44.464730 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:44.464764 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:44.465234 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:44.465464 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:44.465467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:44.465476 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:44.466011 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.466017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.466703 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.466704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.467501 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.467504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.467881 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.467881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.467982 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.468449 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:44.468449 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:44.468475 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:44.468497 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:44.468642 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:44.468644 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:44.468646 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:44.468690 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 14:58:44.468766 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260810 14:58:44.469439 11990 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36151
I20260810 14:58:44.469549 11990 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:44.469568 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 11789
I20260810 14:58:44.469667 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:44.469815 11990 heartbeater.cc:511] Master 127.28.193.254:36151 requested a full tablet report, sending...
I20260810 14:58:44.469934 29447 external_mini_cluster.cc:1428] Setting key 59d97f3ea418f4aa72160f2c6cd48cfb
I20260810 14:58:44.470376 11685 ts_manager.cc:194] Registered new tserver with Master: a12bbd9e4fb04a19824b39c75ae1bfc3 (127.28.193.193:33925)
I20260810 14:58:44.470968 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--builtin_ntp_servers=127.28.193.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:58:44.471030 11685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.193:59601
W20260810 14:58:44.552799 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:44.553016 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:44.553046 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:44.554231 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:44.554275 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:44.554307 11993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:44.555620 11993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:44.555889 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:44.556185 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:44.561172 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.561149 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.561131 12002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.561439 11993 server_base.cc:1034] running on GCE node
I20260810 14:58:44.561653 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:44.561903 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:44.563057 11993 hybrid_clock.cc:648] HybridClock initialized: now 1786373924563048 us; error 32 us; skew 500 ppm
I20260810 14:58:44.564335 11993 webserver.cc:460] Webserver started at http://127.28.193.194:42987/ using document root <none> and password file <none>
I20260810 14:58:44.564599 11993 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:44.564653 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:44.564797 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:44.565922 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "b424ddade965485890e29cbe568a86e5"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "17e6a6faf709852a9e432a50083c6a30"
server_key_iv: "8b0446bef67f3e808de7285d84df82e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.566278 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "b424ddade965485890e29cbe568a86e5"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "17e6a6faf709852a9e432a50083c6a30"
server_key_iv: "8b0446bef67f3e808de7285d84df82e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.567557 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:44.568238 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.568379 11993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:44.568463 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "b424ddade965485890e29cbe568a86e5"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "17e6a6faf709852a9e432a50083c6a30"
server_key_iv: "8b0446bef67f3e808de7285d84df82e6"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.568523 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:44.577033 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:44.577297 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:44.577415 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:44.577625 11993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:44.577965 11993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:44.578001 11993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.578027 11993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:44.578043 11993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.583411 11993 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:38939
I20260810 14:58:44.583477 12215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:38939 every 8 connection(s)
I20260810 14:58:44.583770 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:44.586419 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 11993
I20260810 14:58:44.586495 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:44.586653 29447 external_mini_cluster.cc:1428] Setting key 3dcc8cd0dd23af00b469007a2216401a
I20260810 14:58:44.587628 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--builtin_ntp_servers=127.28.193.212:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:58:44.589001 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.584527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:46671 (local address 127.28.193.254:36151)
0810 14:58:44.584585 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.584587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.584797 (+ 210us) server_negotiation.cc:407] Connection header received
0810 14:58:44.585265 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.585266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.585281 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.585342 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.586044 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.586567 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.587656 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.587927 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.587986 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.588739 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:44.588740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:44.588741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:44.588746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:44.588760 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:44.588764 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:44.588809 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:44.588893 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 14:58:44.588924 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:58:44.589214 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.584452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36151 (local address 127.28.193.194:46671)
0810 14:58:44.584638 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:44.584684 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:44.585167 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:44.585406 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:44.585409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:44.585418 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:44.585960 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.585965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.586652 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.586653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.587575 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.587578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.588003 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.588005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.588124 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.588607 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:44.588608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:44.588635 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:44.588657 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:44.588820 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:44.588825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:44.588828 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 14:58:44.588888 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:58:44.588987 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260810 14:58:44.589653 12216 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36151
I20260810 14:58:44.589775 12216 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:44.589972 12216 heartbeater.cc:511] Master 127.28.193.254:36151 requested a full tablet report, sending...
I20260810 14:58:44.590345 11685 ts_manager.cc:194] Registered new tserver with Master: b424ddade965485890e29cbe568a86e5 (127.28.193.194:38939)
I20260810 14:58:44.590687 11685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.194:46671
W20260810 14:58:44.670670 12224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:44.670941 12224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:44.670967 12224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:44.672123 12224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:44.672163 12224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:44.672188 12224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:44.673502 12224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:44.673753 12224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:44.674067 12224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:44.679066 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.679093 12224 server_base.cc:1034] running on GCE node
W20260810 14:58:44.679059 12233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.679056 12235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.679469 12224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:44.679687 12224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:44.680843 12224 hybrid_clock.cc:648] HybridClock initialized: now 1786373924680832 us; error 33 us; skew 500 ppm
I20260810 14:58:44.682173 12224 webserver.cc:460] Webserver started at http://127.28.193.195:43087/ using document root <none> and password file <none>
I20260810 14:58:44.682384 12224 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:44.682430 12224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:44.682566 12224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:44.683655 12224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "f51bcd7049be4e28ae87e2a0d7bd3018"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "ea549440abe740b6e9e26461b4cb8ba1"
server_key_iv: "d882bb94cbba04878f43ca1b5b2bb089"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.683993 12224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "f51bcd7049be4e28ae87e2a0d7bd3018"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "ea549440abe740b6e9e26461b4cb8ba1"
server_key_iv: "d882bb94cbba04878f43ca1b5b2bb089"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.685243 12224 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:44.686002 12245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.686213 12224 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:44.686308 12224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "f51bcd7049be4e28ae87e2a0d7bd3018"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "ea549440abe740b6e9e26461b4cb8ba1"
server_key_iv: "d882bb94cbba04878f43ca1b5b2bb089"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.686384 12224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:44.714000 12224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:44.714326 12224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:44.714460 12224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:44.714701 12224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:44.715054 12224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:44.715088 12224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.715111 12224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:44.715133 12224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.720590 12224 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:33357
I20260810 14:58:44.720647 12456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:33357 every 8 connection(s)
I20260810 14:58:44.720960 12224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:44.723294 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 12224
I20260810 14:58:44.723395 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:44.723595 29447 external_mini_cluster.cc:1428] Setting key c07ebe6a81cd6a9cc3c84e4b9ee1a18b
I20260810 14:58:44.726243 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.721639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:39923 (local address 127.28.193.254:36151)
0810 14:58:44.721727 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.721730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.721943 (+ 213us) server_negotiation.cc:407] Connection header received
0810 14:58:44.722434 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.722435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.722445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.722505 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.723178 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.723867 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.725002 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.725192 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.725254 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.726027 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:44.726029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:44.726029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:44.726033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:44.726049 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:44.726053 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:44.726101 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:44.726159 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 14:58:44.726182 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:58:44.726424 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.721606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:36151 (local address 127.28.193.195:39923)
0810 14:58:44.721818 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:44.721861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:44.722352 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:44.722577 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:44.722580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:44.722589 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:44.723104 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.723109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.723945 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.723946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.724912 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:44.724915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:44.725262 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:44.725262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:44.725368 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:44.725895 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:44.725896 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:44.725924 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:44.725946 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:44.726105 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:44.726107 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:44.726109 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:44.726152 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:58:44.726228 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":106,"threads_started":1}
I20260810 14:58:44.726890 12459 heartbeater.cc:347] Connected to a master server at 127.28.193.254:36151
I20260810 14:58:44.726996 12459 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:44.727178 12459 heartbeater.cc:511] Master 127.28.193.254:36151 requested a full tablet report, sending...
I20260810 14:58:44.727510 11685 ts_manager.cc:194] Registered new tserver with Master: f51bcd7049be4e28ae87e2a0d7bd3018 (127.28.193.195:33357)
I20260810 14:58:44.727838 11685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.195:39923
I20260810 14:58:44.735162 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:58:44.736837 11787 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:44.736428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.28.193.254:36151)
0810 14:58:44.736490 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.736492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.736569 (+ 77us) server_negotiation.cc:407] Connection header received
0810 14:58:44.736611 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.736612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.736620 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.736663 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.736774 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44664: BlockingRecv error: recv got EOF from 127.0.0.1:44664 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:58:44.736889 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 11789
I20260810 14:58:44.755165 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 11993
I20260810 14:58:44.777242 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 12224
I20260810 14:58:44.794206 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 11640
2026-08-10T14:58:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (660 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T14:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:44Z Disabled control of system clock
I20260810 14:58:44.827996 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33819
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:58:44.909446 12481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:44.909658 12481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:44.909686 12481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:44.910817 12481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:44.910859 12481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:44.910876 12481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:44.910892 12481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:44.910905 12481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:44.912225 12481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:33819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:44.912474 12481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:44.912738 12481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:44.917408 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.917563 12481 server_base.cc:1034] running on GCE node
W20260810 14:58:44.917446 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:44.917413 12492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:44.918061 12481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:44.918341 12481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:44.919508 12481 hybrid_clock.cc:648] HybridClock initialized: now 1786373924919488 us; error 35 us; skew 500 ppm
I20260810 14:58:44.920671 12481 webserver.cc:460] Webserver started at http://127.28.193.254:32891/ using document root <none> and password file <none>
I20260810 14:58:44.920890 12481 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:44.920940 12481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:44.921079 12481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:44.922187 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "bec4cc0eeb32411ab5f25bc460908b46"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "bd3451385a028ab71cecd278d8664798"
server_key_iv: "06e6b208b7b7a8b37001200f5a0503c8"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.922528 12481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "bec4cc0eeb32411ab5f25bc460908b46"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "bd3451385a028ab71cecd278d8664798"
server_key_iv: "06e6b208b7b7a8b37001200f5a0503c8"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.923729 12481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:44.924403 12502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:44.924549 12481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:44.924631 12481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "bec4cc0eeb32411ab5f25bc460908b46"
format_stamp: "Formatted at 2026-08-10 14:58:44 on dist-test-slave-cvwc"
server_key: "bd3451385a028ab71cecd278d8664798"
server_key_iv: "06e6b208b7b7a8b37001200f5a0503c8"
server_key_version: "encryptionkey@0"
I20260810 14:58:44.924690 12481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:44.927894 12481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:44.928125 12481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:44.928246 12481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:44.931890 12481 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:33819
I20260810 14:58:44.931934 12593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:33819 every 8 connection(s)
I20260810 14:58:44.932231 12481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:44.932844 12594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:44.935530 12594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46: Bootstrap starting.
I20260810 14:58:44.936079 12594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:44.936350 12594 log.cc:824] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:44.936982 12594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46: No bootstrap required, opened a new log
I20260810 14:58:44.938588 12594 raft_consensus.cc:348] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } }
I20260810 14:58:44.938723 12594 raft_consensus.cc:374] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:44.938745 12594 raft_consensus.cc:729] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bec4cc0eeb32411ab5f25bc460908b46, State: Initialized, Role: FOLLOWER
I20260810 14:58:44.938884 12594 consensus_queue.cc:260] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } }
I20260810 14:58:44.938951 12594 raft_consensus.cc:388] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:44.938972 12594 raft_consensus.cc:482] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:44.939018 12594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:44.939536 12594 raft_consensus.cc:504] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } }
I20260810 14:58:44.939625 12594 leader_election.cc:302] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bec4cc0eeb32411ab5f25bc460908b46; no voters:
I20260810 14:58:44.939774 12594 leader_election.cc:288] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:44.939821 12598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:44.939992 12598 raft_consensus.cc:686] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [term 1 LEADER]: Becoming Leader. State: Replica: bec4cc0eeb32411ab5f25bc460908b46, State: Running, Role: LEADER
I20260810 14:58:44.940024 12594 sys_catalog.cc:564] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:44.940097 12598 consensus_queue.cc:237] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } }
I20260810 14:58:44.940698 12603 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bec4cc0eeb32411ab5f25bc460908b46. Latest consensus state: current_term: 1 leader_uuid: "bec4cc0eeb32411ab5f25bc460908b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } } }
I20260810 14:58:44.940807 12603 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:44.940949 12600 sys_catalog.cc:455] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bec4cc0eeb32411ab5f25bc460908b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bec4cc0eeb32411ab5f25bc460908b46" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33819 } } }
I20260810 14:58:44.941082 12600 sys_catalog.cc:458] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:44.941069 12620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:44.941610 12620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:44.942070 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 12481
I20260810 14:58:44.942167 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:44.942342 29447 external_mini_cluster.cc:1428] Setting key 971e7b127028a09d36c6f852f24c6db2
I20260810 14:58:44.943259 12620 catalog_manager.cc:1347] Generated new cluster ID: f8fa7952a740413fb39a22da5d9c7527
I20260810 14:58:44.943325 12620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:58:44.944679 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:44.942922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.28.193.254:33819)
0810 14:58:44.943327 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:44.943336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:44.943367 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:58:44.943472 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:44.943477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:44.943511 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:44.943633 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:44.944247 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:44.944251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:44.944251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:44.944255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:44.944293 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:44.944303 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:44.944354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:44.944416 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 14:58:44.944550 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":60,"threads_started":1}
I20260810 14:58:44.964939 12620 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:44.965157 12620 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:44.978348 12620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bec4cc0eeb32411ab5f25bc460908b46: Generated new TSK 0
I20260810 14:58:44.978574 12620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:58:44.981258 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--builtin_ntp_servers=127.28.193.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:58:45.062319 12641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:45.062538 12641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:45.062561 12641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:45.063722 12641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:45.063762 12641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:45.063789 12641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:45.065142 12641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:45.065372 12641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:45.065675 12641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:45.071005 12660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:45.071053 12656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:45.071017 12657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:45.071020 12641 server_base.cc:1034] running on GCE node
I20260810 14:58:45.071362 12641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:45.071614 12641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:45.072749 12641 hybrid_clock.cc:648] HybridClock initialized: now 1786373925072733 us; error 33 us; skew 500 ppm
I20260810 14:58:45.074028 12641 webserver.cc:460] Webserver started at http://127.28.193.193:37413/ using document root <none> and password file <none>
I20260810 14:58:45.074251 12641 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:45.074297 12641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:45.074427 12641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:45.075551 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "5dfdf98c9a3a4e168a4b720179b663ba"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "f8bf1ff1a60fbfe223e6eedc9a571826"
server_key_iv: "53f30e3185032d8a8792975e7880a2cd"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.075896 12641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "5dfdf98c9a3a4e168a4b720179b663ba"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "f8bf1ff1a60fbfe223e6eedc9a571826"
server_key_iv: "53f30e3185032d8a8792975e7880a2cd"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.077184 12641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:45.078038 12676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.078210 12641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:45.078302 12641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "5dfdf98c9a3a4e168a4b720179b663ba"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "f8bf1ff1a60fbfe223e6eedc9a571826"
server_key_iv: "53f30e3185032d8a8792975e7880a2cd"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.078363 12641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:45.081542 12641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:45.081784 12641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:45.081888 12641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:45.082051 12641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:45.082305 12641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:45.082335 12641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.082353 12641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:45.082377 12641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.087651 12641 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:38729
I20260810 14:58:45.087715 12877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:38729 every 8 connection(s)
I20260810 14:58:45.087998 12641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:45.090806 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.088694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:52699 (local address 127.28.193.254:33819)
0810 14:58:45.088756 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:45.088758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:45.088998 (+ 240us) server_negotiation.cc:407] Connection header received
0810 14:58:45.089458 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:45.089459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:45.089465 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:45.089526 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:45.090618 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:45.090619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:45.090620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:45.090624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:45.090639 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:45.090642 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:45.090686 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:45.090736 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:58:45.090748 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:58:45.091008 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.088665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33819 (local address 127.28.193.193:52699)
0810 14:58:45.088881 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:45.088921 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:45.089377 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:45.089579 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:45.089582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:45.089590 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:45.090482 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:45.090482 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:45.090510 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:45.090540 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:45.090686 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:45.090688 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:45.090690 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:45.090738 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 14:58:45.090805 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20260810 14:58:45.091562 12878 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33819
I20260810 14:58:45.091648 12878 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:45.091809 12878 heartbeater.cc:511] Master 127.28.193.254:33819 requested a full tablet report, sending...
I20260810 14:58:45.092314 12528 ts_manager.cc:194] Registered new tserver with Master: 5dfdf98c9a3a4e168a4b720179b663ba (127.28.193.193:38729)
I20260810 14:58:45.096613 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 12641
I20260810 14:58:45.096693 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:45.096874 29447 external_mini_cluster.cc:1428] Setting key d29535db8c2595c809ccc4f6b07d320c
I20260810 14:58:45.097795 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--builtin_ntp_servers=127.28.193.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:58:45.183511 12885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:45.183722 12885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:45.183743 12885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:45.184857 12885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:45.184890 12885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:45.184914 12885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:45.186249 12885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:45.186445 12885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:45.186689 12885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:45.191519 12894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:45.191527 12899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:45.191527 12893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:45.191640 12885 server_base.cc:1034] running on GCE node
I20260810 14:58:45.191918 12885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:45.192126 12885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:45.193254 12885 hybrid_clock.cc:648] HybridClock initialized: now 1786373925193233 us; error 40 us; skew 500 ppm
I20260810 14:58:45.194443 12885 webserver.cc:460] Webserver started at http://127.28.193.194:46359/ using document root <none> and password file <none>
I20260810 14:58:45.194649 12885 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:45.194693 12885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:45.194818 12885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:45.195840 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "f69a5f95320d4cfdbfb52372fa5367bb"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "5ae391ece4ba72f44c54558a4335627d"
server_key_iv: "15186cf1da51f19ac1dd3df9fc0f28fe"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.196169 12885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "f69a5f95320d4cfdbfb52372fa5367bb"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "5ae391ece4ba72f44c54558a4335627d"
server_key_iv: "15186cf1da51f19ac1dd3df9fc0f28fe"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.197441 12885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:45.198310 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.198511 12885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 14:58:45.198606 12885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "f69a5f95320d4cfdbfb52372fa5367bb"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "5ae391ece4ba72f44c54558a4335627d"
server_key_iv: "15186cf1da51f19ac1dd3df9fc0f28fe"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.198676 12885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:45.201867 12885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:45.202105 12885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:45.202234 12885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:45.202428 12885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:45.202702 12885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:45.202734 12885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.202760 12885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:45.202783 12885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.208182 12885 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:41581
I20260810 14:58:45.208235 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:41581 every 8 connection(s)
I20260810 14:58:45.208534 12885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:45.211400 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.209253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:49653 (local address 127.28.193.254:33819)
0810 14:58:45.209319 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:45.209321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:45.209552 (+ 231us) server_negotiation.cc:407] Connection header received
0810 14:58:45.210058 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:45.210059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:45.210064 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:45.210121 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:45.211211 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:45.211213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:45.211213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:45.211218 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:45.211233 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:45.211237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:45.211286 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:45.211336 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:58:45.211347 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:45.211573 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.209235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33819 (local address 127.28.193.194:49653)
0810 14:58:45.209422 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:45.209460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:45.209975 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:45.210175 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:45.210178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:45.210186 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:45.211067 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:45.211067 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:45.211095 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:45.211124 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:45.211290 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:45.211292 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:45.211294 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:45.211341 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 14:58:45.211387 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260810 14:58:45.212008 13102 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33819
I20260810 14:58:45.212090 13102 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:45.212203 13102 heartbeater.cc:511] Master 127.28.193.254:33819 requested a full tablet report, sending...
I20260810 14:58:45.212505 12528 ts_manager.cc:194] Registered new tserver with Master: f69a5f95320d4cfdbfb52372fa5367bb (127.28.193.194:41581)
I20260810 14:58:45.213315 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 12885
I20260810 14:58:45.213388 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:45.213549 29447 external_mini_cluster.cc:1428] Setting key 70c9bbc6ce9058de667e7fa0691f4857
I20260810 14:58:45.214388 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--builtin_ntp_servers=127.28.193.212:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:58:45.295666 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:45.295859 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:45.295889 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:45.296994 13105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:45.297031 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:45.297063 13105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:45.298399 13105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:45.298645 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:45.298914 13105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:45.303670 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:45.303679 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:45.303758 13105 server_base.cc:1034] running on GCE node
W20260810 14:58:45.303972 13117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:45.304159 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:45.304381 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:45.305531 13105 hybrid_clock.cc:648] HybridClock initialized: now 1786373925305511 us; error 29 us; skew 500 ppm
I20260810 14:58:45.306720 13105 webserver.cc:460] Webserver started at http://127.28.193.195:40909/ using document root <none> and password file <none>
I20260810 14:58:45.306948 13105 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:45.306998 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:45.307119 13105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:45.308113 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "23dea8fb6eed4a24814bcd011fa87eb7"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "671d8cbb0668c80d75fd763980527f7f"
server_key_iv: "4abe8ce418e29f905cdd95a76b727619"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.308444 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "23dea8fb6eed4a24814bcd011fa87eb7"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "671d8cbb0668c80d75fd763980527f7f"
server_key_iv: "4abe8ce418e29f905cdd95a76b727619"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.309648 13105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 14:58:45.310302 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.310448 13105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.310527 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "23dea8fb6eed4a24814bcd011fa87eb7"
format_stamp: "Formatted at 2026-08-10 14:58:45 on dist-test-slave-cvwc"
server_key: "671d8cbb0668c80d75fd763980527f7f"
server_key_iv: "4abe8ce418e29f905cdd95a76b727619"
server_key_version: "encryptionkey@0"
I20260810 14:58:45.310588 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:45.313694 13105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:45.313946 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:45.314060 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:45.314273 13105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:45.314553 13105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:45.314585 13105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.314610 13105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:45.314636 13105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:45.319774 13105 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:45037
I20260810 14:58:45.319833 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:45037 every 8 connection(s)
I20260810 14:58:45.320099 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 14:58:45.322835 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.320757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40727 (local address 127.28.193.254:33819)
0810 14:58:45.320800 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:45.320802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:45.321004 (+ 202us) server_negotiation.cc:407] Connection header received
0810 14:58:45.321478 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:45.321479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:45.321487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:45.321539 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:45.322596 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:45.322598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:45.322598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:58:45.322602 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:45.322618 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:58:45.322622 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:45.322675 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:45.322749 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 14:58:45.322764 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:58:45.323019 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:45.320720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33819 (local address 127.28.193.195:40727)
0810 14:58:45.320897 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:45.320929 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:45.321381 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:45.321603 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:45.321607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:45.321616 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:45.322460 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:58:45.322461 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:45.322487 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:58:45.322513 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:45.322676 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:45.322681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:45.322683 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 14:58:45.322758 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:58:45.322831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260810 14:58:45.323436 13306 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33819
I20260810 14:58:45.323529 13306 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:45.323668 13306 heartbeater.cc:511] Master 127.28.193.254:33819 requested a full tablet report, sending...
I20260810 14:58:45.323997 12528 ts_manager.cc:194] Registered new tserver with Master: 23dea8fb6eed4a24814bcd011fa87eb7 (127.28.193.195:45037)
I20260810 14:58:45.329500 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 13105
I20260810 14:58:45.329572 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:45.329763 29447 external_mini_cluster.cc:1428] Setting key 4d37a6912c42e2275fd75c13aa785555
I20260810 14:58:45.330502 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:58:45.332079 12640 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:58:45.331667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.28.193.254:33819)
0810 14:58:45.331710 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:45.331712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:45.331810 (+ 98us) server_negotiation.cc:407] Connection header received
0810 14:58:45.331852 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:45.331853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:45.331860 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:45.331909 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:45.332023 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45972: BlockingRecv error: recv got EOF from 127.0.0.1:45972 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 14:58:45.332160 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 12641
I20260810 14:58:45.343775 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 12885
I20260810 14:58:45.352675 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 13105
I20260810 14:58:45.360792 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 12481
2026-08-10T14:58:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (566 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:45 dist-test-slave-cvwc krb5kdc[13322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:45 dist-test-slave-cvwc krb5kdc[13322](info): set up 2 sockets
Aug 10 14:58:45 dist-test-slave-cvwc krb5kdc[13322](info): commencing operation
krb5kdc: starting...
W20260810 14:58:46.525909 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:46 dist-test-slave-cvwc krb5kdc[13322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:58:46.642208 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46825
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:46.724453 13345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:46.724666 13345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:46.724687 13345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:46.725858 13345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:58:46.725898 13345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:46.725911 13345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:46.725924 13345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:58:46.725946 13345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:58:46.727450 13345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:46.727677 13345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:46.727903 13345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:46.732609 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:46.732583 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:46.732789 13345 server_base.cc:1034] running on GCE node
W20260810 14:58:46.732595 13357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:46.733088 13345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:46.733379 13345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:46.734525 13345 hybrid_clock.cc:648] HybridClock initialized: now 1786373926734513 us; error 30 us; skew 500 ppm
Aug 10 14:58:46 dist-test-slave-cvwc krb5kdc[13322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:46.736948 13345 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:58:46.737350 13345 webserver.cc:460] Webserver started at http://127.28.193.254:35655/ using document root <none> and password file <none>
I20260810 14:58:46.737572 13345 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:46.737622 13345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:46.737779 13345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:46.738837 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "7930327f8e234d5d854fd4753128a2e9"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "2940a5625b1da26550a3500e330b2d46"
server_key_iv: "53e14137881e00bb0fe58dd9012574a1"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.739178 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "7930327f8e234d5d854fd4753128a2e9"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "2940a5625b1da26550a3500e330b2d46"
server_key_iv: "53e14137881e00bb0fe58dd9012574a1"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.740489 13345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:58:46.741150 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:46.741297 13345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:46.741390 13345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "7930327f8e234d5d854fd4753128a2e9"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "2940a5625b1da26550a3500e330b2d46"
server_key_iv: "53e14137881e00bb0fe58dd9012574a1"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.741462 13345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:46.755733 13345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:46.757625 13345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:46.757789 13345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:46.761508 13345 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:46825
I20260810 14:58:46.761569 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:46825 every 8 connection(s)
I20260810 14:58:46.761906 13345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:58:46.762907 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:46.765192 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: Bootstrap starting.
I20260810 14:58:46.765790 13446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:46.766036 13446 log.cc:824] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:46.766697 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: No bootstrap required, opened a new log
I20260810 14:58:46.767052 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 13345
I20260810 14:58:46.767139 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:58:46.767292 29447 external_mini_cluster.cc:1428] Setting key 036a8f487137884f7a897a241921076c
I20260810 14:58:46.768532 13446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } }
I20260810 14:58:46.768697 13446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:46.768735 13446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7930327f8e234d5d854fd4753128a2e9, State: Initialized, Role: FOLLOWER
I20260810 14:58:46.768880 13446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } }
I20260810 14:58:46.768966 13446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:58:46.769004 13446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:58:46.769053 13446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:46.770021 13446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } }
I20260810 14:58:46.770114 13446 leader_election.cc:302] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7930327f8e234d5d854fd4753128a2e9; no voters:
I20260810 14:58:46.770274 13446 leader_election.cc:288] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:58:46.770318 13454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:46.770473 13454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7930327f8e234d5d854fd4753128a2e9, State: Running, Role: LEADER
I20260810 14:58:46.770543 13446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:58:46.770642 13454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } }
I20260810 14:58:46.770987 13454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7930327f8e234d5d854fd4753128a2e9. Latest consensus state: current_term: 1 leader_uuid: "7930327f8e234d5d854fd4753128a2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } } }
I20260810 14:58:46.771230 13454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:46.771163 13458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7930327f8e234d5d854fd4753128a2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7930327f8e234d5d854fd4753128a2e9" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 46825 } } }
I20260810 14:58:46.771661 13458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:58:46.771711 13476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:58:46.772280 13476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:58:46.773804 13476 catalog_manager.cc:1347] Generated new cluster ID: a250d43430054c65af88ef066822c883
I20260810 14:58:46.773854 13476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 14:58:46 dist-test-slave-cvwc krb5kdc[13322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:46.778599 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:46.767793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.28.193.254:46825)
0810 14:58:46.767956 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:46.767965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:46.767990 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:58:46.768494 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:46.768500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:46.768545 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:46.768654 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:46.769431 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.770505 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.771528 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.772214 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.772325 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:46.775093 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:46.775097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:46.775097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:46.775109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:46.776948 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:46.777332 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:46.777335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:46.777335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:46.777460 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:46.777658 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:46.777658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:46.777659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:46.777965 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:46.778116 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:46.778246 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 14:58:46.778374 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260810 14:58:46.787585 13476 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:58:46.788084 13476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:58:46.792526 13476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: Generated new TSK 0
I20260810 14:58:46.792690 13476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:58:46.821833 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:46.905813 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:46.906020 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:46.906042 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:46.907204 13498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:46.907245 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:46.907271 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:58:46.908692 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:46.908905 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:46.909122 13498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:46.914011 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:46.914041 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:46.914072 13498 server_base.cc:1034] running on GCE node
W20260810 14:58:46.914038 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:46.914377 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:46.914641 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:46.915808 13498 hybrid_clock.cc:648] HybridClock initialized: now 1786373926915793 us; error 30 us; skew 500 ppm
Aug 10 14:58:46 dist-test-slave-cvwc krb5kdc[13322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:46.918417 13498 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:58:46.918852 13498 webserver.cc:460] Webserver started at http://127.28.193.193:37555/ using document root <none> and password file <none>
I20260810 14:58:46.919075 13498 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:46.919123 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:46.919247 13498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:46.920306 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "d816f488a6034945b3da972e80521ab6"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "dfaffeeefeb2c6459f1e499dea2d0484"
server_key_iv: "93f5694d74860bea892497c9c7c84f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.920652 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "d816f488a6034945b3da972e80521ab6"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "dfaffeeefeb2c6459f1e499dea2d0484"
server_key_iv: "93f5694d74860bea892497c9c7c84f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.922000 13498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:46.922765 13518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:46.922959 13498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:58:46.923053 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "d816f488a6034945b3da972e80521ab6"
format_stamp: "Formatted at 2026-08-10 14:58:46 on dist-test-slave-cvwc"
server_key: "dfaffeeefeb2c6459f1e499dea2d0484"
server_key_iv: "93f5694d74860bea892497c9c7c84f35"
server_key_version: "encryptionkey@0"
I20260810 14:58:46.923115 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:46.953944 13498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:46.956012 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:46.956154 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:46.956401 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:46.956708 13498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:46.956743 13498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:46.956772 13498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:46.956789 13498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:46.962294 13498 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:45877
I20260810 14:58:46.962338 13714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:45877 every 8 connection(s)
I20260810 14:58:46.962666 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 14:58:46.967083 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 13498
I20260810 14:58:46.967188 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:58:46.967404 29447 external_mini_cluster.cc:1428] Setting key f585d4c4d498ec6fb53463b7c0072eae
Aug 10 14:58:46 dist-test-slave-cvwc krb5kdc[13322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373926, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:46.972128 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:46.963459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38351 (local address 127.28.193.254:46825)
0810 14:58:46.963536 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:46.963538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:46.963709 (+ 171us) server_negotiation.cc:407] Connection header received
0810 14:58:46.964215 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:46.964216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:46.964227 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:46.964298 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:46.965262 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.965806 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.966776 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.966978 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.967023 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:46.969466 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:46.969467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:46.969467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:46.969476 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:46.970796 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:46.971154 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:46.971156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:46.971156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:46.971221 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:46.971484 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:46.971485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:46.971485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:46.971604 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:46.971660 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:46.971953 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 14:58:46.972039 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:58:46.972287 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:46.963418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46825 (local address 127.28.193.193:38351)
0810 14:58:46.963582 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:46.963625 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:46.964126 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:46.964356 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:46.964359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:46.964948 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:46.965144 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:46.965149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.965891 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.965892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:46.966699 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:46.966702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:46.967052 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:46.967054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:46.967209 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:46.967830 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:46.967838 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:46.969376 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:46.970865 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:46.970871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:46.970873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:46.971078 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:46.971284 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:46.971285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:46.971285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:46.971425 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:46.971660 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:46.971668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:46.971828 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 14:58:46.971919 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:58:46.972070 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260810 14:58:46.972740 13715 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46825
I20260810 14:58:46.972834 13715 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:46.973006 13715 heartbeater.cc:511] Master 127.28.193.254:46825 requested a full tablet report, sending...
I20260810 14:58:46.973516 13392 ts_manager.cc:194] Registered new tserver with Master: d816f488a6034945b3da972e80521ab6 (127.28.193.193:45877)
I20260810 14:58:46.974159 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:38351
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:58:46.992213 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:47.073364 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:47.073566 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:47.073587 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:47.074712 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:47.074755 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:47.074779 13725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:58:47.076206 13725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:47.076438 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:47.076660 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:47.081593 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:47.081561 13734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:47.081576 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:47.081750 13725 server_base.cc:1034] running on GCE node
I20260810 14:58:47.082034 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:47.082257 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:47.083434 13725 hybrid_clock.cc:648] HybridClock initialized: now 1786373927083417 us; error 36 us; skew 500 ppm
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[13322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:47.085966 13725 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:58:47.086383 13725 webserver.cc:460] Webserver started at http://127.28.193.194:45925/ using document root <none> and password file <none>
I20260810 14:58:47.086596 13725 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:47.086637 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:47.086730 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:47.087769 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "0f3fd3cdae7546f189e0643dc685823e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "d6c3323a0d31ad1f2ff5aa75ebd24a91"
server_key_iv: "0a5b2e0e50129170a5e0b67876924a48"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.088083 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "0f3fd3cdae7546f189e0643dc685823e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "d6c3323a0d31ad1f2ff5aa75ebd24a91"
server_key_iv: "0a5b2e0e50129170a5e0b67876924a48"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.089291 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 14:58:47.089984 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.090121 13725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 14:58:47.090207 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "0f3fd3cdae7546f189e0643dc685823e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "d6c3323a0d31ad1f2ff5aa75ebd24a91"
server_key_iv: "0a5b2e0e50129170a5e0b67876924a48"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.090276 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:47.103565 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:47.105638 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:47.105875 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:47.106081 13725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:47.106367 13725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:47.106410 13725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.106431 13725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:47.106446 13725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.112087 13725 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:42183
I20260810 14:58:47.112128 13950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:42183 every 8 connection(s)
I20260810 14:58:47.112445 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:58:47.117365 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 13725
I20260810 14:58:47.117460 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:58:47.117630 29447 external_mini_cluster.cc:1428] Setting key fce91810271b873505df805fc1f860bb
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[13322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:47.122176 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.113152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:34841 (local address 127.28.193.254:46825)
0810 14:58:47.113208 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.113210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.113420 (+ 210us) server_negotiation.cc:407] Connection header received
0810 14:58:47.114016 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.114017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.114029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.114094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:47.115123 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.115647 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.116783 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.116987 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.117045 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.119462 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:47.119463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:47.119464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:47.119471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:47.120759 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.121210 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.121211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.121212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.121279 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.121490 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.121490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.121491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.121622 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:47.121682 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.122003 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.122090 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:58:47.122262 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.113119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46825 (local address 127.28.193.194:34841)
0810 14:58:47.113286 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.113329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.113891 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.114203 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.114207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.114809 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:47.115012 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.115017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.115738 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.115739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.116691 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.116695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.117059 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.117059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.117193 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.117793 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:47.117804 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:47.119370 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:47.120839 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:47.120845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:47.120848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:47.121144 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:47.121338 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:47.121338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:47.121338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:47.121437 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:47.121680 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:47.121683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:47.121872 (+ 189us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.121960 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.122083 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260810 14:58:47.122733 13951 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46825
I20260810 14:58:47.122848 13951 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:47.123027 13951 heartbeater.cc:511] Master 127.28.193.254:46825 requested a full tablet report, sending...
I20260810 14:58:47.123420 13392 ts_manager.cc:194] Registered new tserver with Master: 0f3fd3cdae7546f189e0643dc685823e (127.28.193.194:42183)
I20260810 14:58:47.123919 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:34841
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:58:47.142351 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:58:47.223136 13961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:58:47.223344 13961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:58:47.223374 13961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:58:47.224501 13961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:58:47.224541 13961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:58:47.224575 13961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:58:47.226058 13961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:46825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:58:47.226296 13961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:58:47.226512 13961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:58:47.231261 13977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:47.231334 13970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:58:47.231287 13971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:58:47.231487 13961 server_base.cc:1034] running on GCE node
I20260810 14:58:47.231673 13961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:58:47.231880 13961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:58:47.233027 13961 hybrid_clock.cc:648] HybridClock initialized: now 1786373927233012 us; error 28 us; skew 500 ppm
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[13322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:58:47.235455 13961 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:58:47.235875 13961 webserver.cc:460] Webserver started at http://127.28.193.195:39159/ using document root <none> and password file <none>
I20260810 14:58:47.236076 13961 fs_manager.cc:359] Metadata directory not provided
I20260810 14:58:47.236122 13961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:58:47.236230 13961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:58:47.237272 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "05d8284a12954120af4e4dbe01ba943e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "e9650612169ace12126b6503c55a371b"
server_key_iv: "e19a7901467a3d6b84a6b47b6c737a56"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.237610 13961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "05d8284a12954120af4e4dbe01ba943e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "e9650612169ace12126b6503c55a371b"
server_key_iv: "e19a7901467a3d6b84a6b47b6c737a56"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.238900 13961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:58:47.239590 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.239744 13961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 14:58:47.239825 13961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "05d8284a12954120af4e4dbe01ba943e"
format_stamp: "Formatted at 2026-08-10 14:58:47 on dist-test-slave-cvwc"
server_key: "e9650612169ace12126b6503c55a371b"
server_key_iv: "e19a7901467a3d6b84a6b47b6c737a56"
server_key_version: "encryptionkey@0"
I20260810 14:58:47.239888 13961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:58:47.290314 13961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:58:47.292096 13961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:58:47.292233 13961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:58:47.292518 13961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:58:47.292896 13961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:58:47.292932 13961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.292954 13961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:58:47.292968 13961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:58:47.298607 13961 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:46397
I20260810 14:58:47.298660 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:46397 every 8 connection(s)
I20260810 14:58:47.298969 13961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[13322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373927, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:58:47.307395 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.299666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:33151 (local address 127.28.193.254:46825)
0810 14:58:47.299721 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.299723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.299900 (+ 177us) server_negotiation.cc:407] Connection header received
0810 14:58:47.300373 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.300374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.300383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.300440 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:47.301353 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.301880 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.302791 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.302975 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.303017 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.304932 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:47.304933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:47.304934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:47.304940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:47.306189 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.306558 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.306559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.306560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.306622 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.306861 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.306862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.306862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.306999 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:47.307064 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.307267 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.307331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 14:58:47.307569 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.299627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:46825 (local address 127.28.193.195:33151)
0810 14:58:47.299784 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.299826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.300301 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.300502 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.300505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.301088 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:58:47.301283 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.301288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.301944 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.301944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.302726 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.302728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.303026 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.303027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.303139 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.303565 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:58:47.303574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:58:47.304839 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:58:47.306255 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:47.306259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:47.306261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:47.306474 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:47.306680 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:58:47.306681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:58:47.306681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:58:47.306777 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:58:47.307057 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:58:47.307060 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:58:47.307172 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.307246 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.307364 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260810 14:58:47.308065 14189 heartbeater.cc:347] Connected to a master server at 127.28.193.254:46825
I20260810 14:58:47.308157 14189 heartbeater.cc:464] Registering TS with master...
I20260810 14:58:47.308358 14189 heartbeater.cc:511] Master 127.28.193.254:46825 requested a full tablet report, sending...
I20260810 14:58:47.308395 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 13961
I20260810 14:58:47.308486 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:58:47.308641 29447 external_mini_cluster.cc:1428] Setting key c34f2c383cb0e43838414f29ef701d31
I20260810 14:58:47.308787 13392 ts_manager.cc:194] Registered new tserver with Master: 05d8284a12954120af4e4dbe01ba943e (127.28.193.195:46397)
I20260810 14:58:47.309197 13392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:33151
I20260810 14:58:47.309428 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:58:47.315899 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.310727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.28.193.254:46825)
0810 14:58:47.310780 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.310782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.310821 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:58:47.310870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.310871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.310882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.310959 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:58:47.311412 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.311881 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.312517 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.312684 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.312752 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.313695 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:58:47.313697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:58:47.313697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:58:47.313703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:58:47.315054 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.315317 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.315317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.315318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.315366 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:58:47.315498 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:58:47.315499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:58:47.315499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:58:47.315630 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:58:47.315702 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.315769 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.315839 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 14:58:47.317379 13392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:58:47.318984 13392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:58:47.325153 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:46397 (local address 127.0.0.1:59574)
0810 14:58:47.321593 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.321602 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.321661 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.321863 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.321865 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.321867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:47.322049 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.322051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.322902 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.322903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.324130 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.324132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.324816 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.324894 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.324907 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.324980 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.325035 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260810 14:58:47.325213 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42183 (local address 127.0.0.1:51656)
0810 14:58:47.321744 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.321761 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.321806 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.322077 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.322078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.322080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:47.322114 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.322116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.323313 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.323314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.324131 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.324133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.324981 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.325035 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.325039 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.325090 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.325121 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260810 14:58:47.325268 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.28.193.194:42183)
0810 14:58:47.321583 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.321588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.321861 (+ 273us) server_negotiation.cc:407] Connection header received
0810 14:58:47.321907 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.321910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.321926 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.321999 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:47.322184 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.323226 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324202 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.324900 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324974 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.325027 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.325110 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.325186 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260810 14:58:47.325480 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.28.193.195:46397)
0810 14:58:47.321553 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.321558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.321663 (+ 105us) server_negotiation.cc:407] Connection header received
0810 14:58:47.321736 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.321738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.321752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.321912 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:47.322124 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.322806 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324223 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.324740 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324805 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.324865 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.325365 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.325406 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":86,"threads_started":1}
I20260810 14:58:47.325503 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45877 (local address 127.0.0.1:36748)
0810 14:58:47.321320 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.321357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.321423 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.321791 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.321795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.321798 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:47.321882 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.321887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.323216 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.323217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.324253 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.324255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.325093 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.325095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.325185 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.325191 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.325420 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.325456 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260810 14:58:47.325888 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.321411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.28.193.193:45877)
0810 14:58:47.321584 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.321588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.321597 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:58:47.321638 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.321640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.321655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.321760 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:47.321965 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.323117 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.324325 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.325001 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.325660 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.325743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.325768 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.325809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260810 14:58:47.326296 14074 tablet_service.cc:1467] Processing CreateTablet for tablet 5afb29b0c1534e7383094f38b9e04278 (DEFAULT_TABLE table=test-table [id=dd6ea4c33ec84be2b9682dff6d57c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:47.326296 13834 tablet_service.cc:1467] Processing CreateTablet for tablet 5afb29b0c1534e7383094f38b9e04278 (DEFAULT_TABLE table=test-table [id=dd6ea4c33ec84be2b9682dff6d57c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:47.326588 13590 tablet_service.cc:1467] Processing CreateTablet for tablet 5afb29b0c1534e7383094f38b9e04278 (DEFAULT_TABLE table=test-table [id=dd6ea4c33ec84be2b9682dff6d57c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:58:47.326680 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afb29b0c1534e7383094f38b9e04278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:47.326680 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afb29b0c1534e7383094f38b9e04278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:47.326885 13590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afb29b0c1534e7383094f38b9e04278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:58:47.329337 14212 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: Bootstrap starting.
I20260810 14:58:47.329854 14213 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Bootstrap starting.
I20260810 14:58:47.330106 14212 tablet_bootstrap.cc:654] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:47.330502 14213 tablet_bootstrap.cc:654] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:47.330669 14212 log.cc:824] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:47.331009 14213 log.cc:824] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:47.331287 14214 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Bootstrap starting.
I20260810 14:58:47.331419 14212 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: No bootstrap required, opened a new log
I20260810 14:58:47.331504 14212 ts_tablet_manager.cc:1397] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:58:47.331686 14213 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: No bootstrap required, opened a new log
I20260810 14:58:47.331753 14213 ts_tablet_manager.cc:1397] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:47.331866 14214 tablet_bootstrap.cc:654] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Neither blocks nor log segments found. Creating new log.
I20260810 14:58:47.332288 14214 log.cc:824] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:58:47.332927 14214 tablet_bootstrap.cc:492] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: No bootstrap required, opened a new log
I20260810 14:58:47.332985 14214 ts_tablet_manager.cc:1397] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:58:47.333397 14212 raft_consensus.cc:348] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.333586 14212 raft_consensus.cc:374] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:47.333513 14213 raft_consensus.cc:348] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.333622 14212 raft_consensus.cc:729] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d816f488a6034945b3da972e80521ab6, State: Initialized, Role: FOLLOWER
I20260810 14:58:47.333643 14213 raft_consensus.cc:374] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:47.333674 14213 raft_consensus.cc:729] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d8284a12954120af4e4dbe01ba943e, State: Initialized, Role: FOLLOWER
I20260810 14:58:47.333791 14213 consensus_queue.cc:260] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.333792 14212 consensus_queue.cc:260] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.334080 14213 ts_tablet_manager.cc:1428] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 14:58:47.334080 14212 ts_tablet_manager.cc:1428] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:58:47.334103 14189 heartbeater.cc:503] Master 127.28.193.254:46825 was elected leader, sending a full tablet report...
I20260810 14:58:47.334218 13715 heartbeater.cc:503] Master 127.28.193.254:46825 was elected leader, sending a full tablet report...
I20260810 14:58:47.334623 14214 raft_consensus.cc:348] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.334771 14214 raft_consensus.cc:374] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:58:47.334806 14214 raft_consensus.cc:729] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3fd3cdae7546f189e0643dc685823e, State: Initialized, Role: FOLLOWER
I20260810 14:58:47.334911 14214 consensus_queue.cc:260] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.335184 14214 ts_tablet_manager.cc:1428] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:58:47.335314 13951 heartbeater.cc:503] Master 127.28.193.254:46825 was elected leader, sending a full tablet report...
W20260810 14:58:47.363005 13954 tablet.cc:2369] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:58:47.400611 14221 raft_consensus.cc:482] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:58:47.400758 14221 raft_consensus.cc:504] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.401149 14221 leader_election.cc:288] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05d8284a12954120af4e4dbe01ba943e (127.28.193.195:46397), 0f3fd3cdae7546f189e0643dc685823e (127.28.193.194:42183)
I20260810 14:58:47.404196 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.401324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:46397 (local address 127.28.193.193:58669)
0810 14:58:47.401403 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.401415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.401474 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.401748 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.401749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.401751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:47.401819 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.401821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.402425 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.402426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.403363 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.403365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.404004 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.404005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.404074 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.404078 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.404113 (+ 35us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.404143 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:58:47.404238 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.401575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:58669 (local address 127.28.193.195:46397)
0810 14:58:47.401618 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.401620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.401627 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:58:47.401652 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.401652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.401659 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.401704 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:47.401882 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.402347 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.403441 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.403916 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.403987 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.403997 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.404136 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.404171 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 14:58:47.404502 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afb29b0c1534e7383094f38b9e04278" candidate_uuid: "d816f488a6034945b3da972e80521ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d8284a12954120af4e4dbe01ba943e" is_pre_election: true
I20260810 14:58:47.404641 14107 raft_consensus.cc:2454] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d816f488a6034945b3da972e80521ab6 in term 0.
I20260810 14:58:47.404857 13519 leader_election.cc:302] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d8284a12954120af4e4dbe01ba943e, d816f488a6034945b3da972e80521ab6; no voters:
I20260810 14:58:47.404975 14221 raft_consensus.cc:2781] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:58:47.405030 14221 raft_consensus.cc:482] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:58:47.405052 14221 raft_consensus.cc:3037] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:47.405650 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.401542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:56615 (local address 127.28.193.194:42183)
0810 14:58:47.401589 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.401591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.402949 (+ 1358us) server_negotiation.cc:407] Connection header received
0810 14:58:47.402985 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.402986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.402995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.403044 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:58:47.403160 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.403594 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.404733 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.405320 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.405393 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.405400 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.405554 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.405587 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 14:58:47.405654 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.401343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42183 (local address 127.28.193.193:56615)
0810 14:58:47.402867 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.402879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:58:47.402942 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:58:47.403049 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:58:47.403049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:58:47.403051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:58:47.403110 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.403112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.403664 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.403665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.404643 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:58:47.404646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.405408 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.405409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:58:47.405479 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.405483 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:58:47.405543 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:58:47.405581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1505,"thread_start_us":1442,"threads_started":1}
I20260810 14:58:47.405939 14221 raft_consensus.cc:504] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.406078 14221 leader_election.cc:288] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [CANDIDATE]: Term 1 election: Requested vote from peers 05d8284a12954120af4e4dbe01ba943e (127.28.193.195:46397), 0f3fd3cdae7546f189e0643dc685823e (127.28.193.194:42183)
I20260810 14:58:47.406045 13870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afb29b0c1534e7383094f38b9e04278" candidate_uuid: "d816f488a6034945b3da972e80521ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3fd3cdae7546f189e0643dc685823e" is_pre_election: true
I20260810 14:58:47.406190 13870 raft_consensus.cc:2454] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d816f488a6034945b3da972e80521ab6 in term 0.
I20260810 14:58:47.406215 13869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afb29b0c1534e7383094f38b9e04278" candidate_uuid: "d816f488a6034945b3da972e80521ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f3fd3cdae7546f189e0643dc685823e"
I20260810 14:58:47.406261 14107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afb29b0c1534e7383094f38b9e04278" candidate_uuid: "d816f488a6034945b3da972e80521ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d8284a12954120af4e4dbe01ba943e"
I20260810 14:58:47.406298 13869 raft_consensus.cc:3037] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:47.406337 14107 raft_consensus.cc:3037] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:58:47.406989 14107 raft_consensus.cc:2454] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d816f488a6034945b3da972e80521ab6 in term 1.
I20260810 14:58:47.407024 13869 raft_consensus.cc:2454] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d816f488a6034945b3da972e80521ab6 in term 1.
I20260810 14:58:47.407140 13519 leader_election.cc:302] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d8284a12954120af4e4dbe01ba943e, d816f488a6034945b3da972e80521ab6; no voters:
I20260810 14:58:47.407222 14221 raft_consensus.cc:2781] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:58:47.407366 14221 raft_consensus.cc:686] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 1 LEADER]: Becoming Leader. State: Replica: d816f488a6034945b3da972e80521ab6, State: Running, Role: LEADER
I20260810 14:58:47.407472 14221 consensus_queue.cc:237] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } }
I20260810 14:58:47.408358 13392 catalog_manager.cc:5495] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d816f488a6034945b3da972e80521ab6 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "d816f488a6034945b3da972e80521ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d816f488a6034945b3da972e80521ab6" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 45877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 } health_report { overall_health: UNKNOWN } } }
I20260810 14:58:47.415800 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.413864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.28.193.193:45877)
0810 14:58:47.413900 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 14:58:47.413902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:58:47.413930 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:58:47.413957 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:58:47.413957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:58:47.413972 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:58:47.414019 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:58:47.414097 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.414539 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.415204 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:58:47.415397 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:58:47.415447 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:58:47.415554 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:58:47.415648 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:58:47.415702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 14:58:47.415718 (+ 16us) server_negotiation.cc:299] Negotiation successful
0810 14:58:47.415733 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 14:58:47.417510 14107 raft_consensus.cc:1261] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 1 FOLLOWER]: Refusing update from remote peer d816f488a6034945b3da972e80521ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:47.417513 13869 raft_consensus.cc:1261] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 1 FOLLOWER]: Refusing update from remote peer d816f488a6034945b3da972e80521ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:58:47.417832 14221 consensus_queue.cc:1035] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d8284a12954120af4e4dbe01ba943e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:47.417966 14228 consensus_queue.cc:1035] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f3fd3cdae7546f189e0643dc685823e" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:58:47.420078 14232 mvcc.cc:205] Tried to move back new op lower bound from 7316987606699569152 to 7316987606661525504. Current Snapshot: MvccSnapshot[applied={T|T < 7316987606699569152}]
I20260810 14:58:47.420261 14234 mvcc.cc:205] Tried to move back new op lower bound from 7316987606699569152 to 7316987606661525504. Current Snapshot: MvccSnapshot[applied={T|T < 7316987606699569152}]
I20260810 14:58:47.422482 13389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:47.422564 13389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:58:47.423237 13389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7930327f8e234d5d854fd4753128a2e9: Sending DeleteTablet for 3 replicas of tablet 5afb29b0c1534e7383094f38b9e04278
I20260810 14:58:47.423534 14074 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afb29b0c1534e7383094f38b9e04278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:59574
I20260810 14:58:47.423594 13834 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afb29b0c1534e7383094f38b9e04278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:51656
I20260810 14:58:47.423632 13590 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afb29b0c1534e7383094f38b9e04278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:58:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:36748
I20260810 14:58:47.423791 14250 tablet_replica.cc:330] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6: stopping tablet replica
I20260810 14:58:47.423877 14250 raft_consensus.cc:2229] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:58:47.423956 14250 raft_consensus.cc:2258] T 5afb29b0c1534e7383094f38b9e04278 P d816f488a6034945b3da972e80521ab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:47.424155 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 13498
I20260810 14:58:47.436702 14251 tablet_replica.cc:330] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: stopping tablet replica
I20260810 14:58:47.436861 14251 raft_consensus.cc:2229] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:47.436968 14251 raft_consensus.cc:2258] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:47.437911 14251 ts_tablet_manager.cc:1905] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:47.440265 14251 ts_tablet_manager.cc:1918] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:47.440325 14251 log.cc:1196] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/5afb29b0c1534e7383094f38b9e04278
I20260810 14:58:47.440528 14251 ts_tablet_manager.cc:1939] T 5afb29b0c1534e7383094f38b9e04278 P 0f3fd3cdae7546f189e0643dc685823e: Deleting consensus metadata
I20260810 14:58:47.440943 13371 catalog_manager.cc:4841] TS 0f3fd3cdae7546f189e0643dc685823e (127.28.193.194:42183): tablet 5afb29b0c1534e7383094f38b9e04278 (table test-table [id=dd6ea4c33ec84be2b9682dff6d57c9ce]) successfully deleted
I20260810 14:58:47.455077 14249 tablet_replica.cc:330] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: stopping tablet replica
I20260810 14:58:47.455246 14249 raft_consensus.cc:2229] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:58:47.455384 14249 raft_consensus.cc:2258] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:58:47.455920 14249 ts_tablet_manager.cc:1905] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:58:47.457576 14249 ts_tablet_manager.cc:1918] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:58:47.457677 14249 log.cc:1196] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/5afb29b0c1534e7383094f38b9e04278
I20260810 14:58:47.457991 14249 ts_tablet_manager.cc:1939] T 5afb29b0c1534e7383094f38b9e04278 P 05d8284a12954120af4e4dbe01ba943e: Deleting consensus metadata
I20260810 14:58:47.458516 13368 catalog_manager.cc:4841] TS 05d8284a12954120af4e4dbe01ba943e (127.28.193.195:46397): tablet 5afb29b0c1534e7383094f38b9e04278 (table test-table [id=dd6ea4c33ec84be2b9682dff6d57c9ce]) successfully deleted
W20260810 14:58:47.459913 13369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:45877 (error 108)
I20260810 14:58:47.460137 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 13725
I20260810 14:58:47.460460 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:58:47.460338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:45877 (local address 127.0.0.1:36762)
0810 14:58:47.460389 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 14:58:47.460422 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 14:58:47.460592 13369 catalog_manager.cc:4584] TS d816f488a6034945b3da972e80521ab6 (127.28.193.193:45877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5afb29b0c1534e7383094f38b9e04278: Network error: Client connection negotiation failed: client connection to 127.28.193.193:45877: connect: Connection refused (error 111)
I20260810 14:58:47.470149 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 13961
I20260810 14:58:47.481889 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 13345
2026-08-10T14:58:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2122 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[14257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[14257](info): set up 2 sockets
Aug 10 14:58:47 dist-test-slave-cvwc krb5kdc[14257](info): commencing operation
krb5kdc: starting...
W20260810 14:58:48.676909 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:58:48 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:58:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangeradmin_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangeradmin_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangerlookup_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangerlookup_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
I20260810 14:58:48.834481 29447 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskILwtmC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/postgres -l logfile start
2026-08-10 14:58:51.206 UTC [14299] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:58:51.206 UTC [14299] LOG: listening on IPv4 address "127.28.193.212", port 53349
2026-08-10 14:58:51.210 UTC [14299] LOG: listening on Unix socket "/tmp/.s.PGSQL.53349"
2026-08-10 14:58:51.238 UTC [14305] LOG: database system was shut down at 2026-08-10 14:58:49 UTC
2026-08-10 14:58:51.242 UTC [14299] LOG: database system is ready to accept connections
I20260810 14:58:52.309371 29447 mini_postgres.cc:96] Postgres bound to 53349
2026-08-10 14:58:52.318 UTC [14318] FATAL: database "slave" does not exist
127.28.193.212:53349 - accepting connections
I20260810 14:58:52.321972 29447 mini_ranger.cc:161] Starting Ranger...
I20260810 14:58:52.338557 29447 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:58:52.766088 29447 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:58:52.766160 29447 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin
2026-08-10 14:58:53,066 [I] DB FLAVOR :POSTGRES
2026-08-10 14:58:53,066 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:58:53,067 [I] Checking connection..
2026-08-10 14:58:53,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:58:53,331 [I] Checking connection passed.
2026-08-10 14:58:53,331 [I] --------- Verifying version history table ---------
2026-08-10 14:58:53,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:58:53,587 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:58:53,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:58:53,951 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:58:53,951 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:58:53,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:58:54.396 UTC [14434] WARNING: there is no transaction in progress
2026-08-10 14:58:54,406 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:58:54,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:58:54,842 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:58:54,843 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:58:54,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:58:55,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:58:55,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-10 14:58:55,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:58:56,018 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:58:56,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:58:56,272 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:58:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:58:56,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:58:56,771 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:58:56,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:58:57.685 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.701 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.716 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.731 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.747 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.767 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.830 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.836 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.849 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.859 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.869 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.882 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.891 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.897 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.903 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:57.909 UTC [14622] WARNING: there is no transaction in progress
2026-08-10 14:58:58,510 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:58:58,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:58:58,745 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:58:58,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:58:59,008 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:58:59,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:58:59,250 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:58:59,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:58:59,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:58:59,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-10 14:58:59,999 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:59:00,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:59:00,265 [I] DB_PATCHES have already been applied
I20260810 14:59:00.271090 29447 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskILwtmC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:59:00.271157 29447 mini_ranger.cc:193] Using host: 127.28.193.212
I20260810 14:59:00.274791 29447 mini_ranger.cc:227] Ranger admin URL: http://127.28.193.212:36197
Aug 10, 2026 2:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:59:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:59:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:59:01.541090 29447 mini_ranger.cc:160] Time spent starting Ranger: real 9.219s user 0.003s sys 0.008s
Aug 10, 2026 2:59:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 2:59:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:59:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:59:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:59:22.967294 29447 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangerkms_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/rangerkms_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.193.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.193.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:59:23 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:59:23.051590 29447 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:59:23.060721 29447 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:59:23.800773 29447 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:59:23.800859 29447 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms
I20260810 14:59:23.800875 29447 mini_ranger_kms.cc:227] Using postgres at 127.28.193.212:53349
2026-08-10 14:59:24,197 [I] DB FLAVOR :POSTGRES
2026-08-10 14:59:24,197 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:59:24,197 [I] Checking connection
2026-08-10 14:59:24,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:59:24,434 [I] connection success
2026-08-10 14:59:24,434 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:59:24,434 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:59:24,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:59:24,695 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:59:24,696 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:59:24,696 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:59:24,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:53349/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:59:24,954 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:59:25.397622 29447 mini_ranger_kms.cc:310] Created kms service
I20260810 14:59:25.573171 29447 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:59:25.617388 29447 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:59:25 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:25.999680 29447 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:59:25.999805 29447 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms:/tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:59:25.999828 29447 mini_ranger_kms.cc:240] Using host: 127.28.193.212
I20260810 14:59:26.004514 29447 mini_ranger_kms.cc:276] Ranger KMS PID: 14918
I20260810 14:59:26.004626 29447 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.193.212:38905
Aug 10, 2026 2:59:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:59:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:59:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 14:59:28 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786373968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:59:28.512841 29447 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.461s user 0.006s sys 0.223s
I20260810 14:59:28.512938 29447 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:59:28.513023 29447 mini_ranger_kms.cc:401] 127.28.193.212:38905/kms/v1/keys
Aug 10 14:59:28 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
Aug 10 14:59:29 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
Aug 10 14:59:31 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 14:59:31.371928 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37847
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:37847
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:38905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:59:31.478561 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:59:31.478760 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:59:31.478780 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:59:31.479908 14992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:59:31.479950 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:59:31.479964 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:59:31.479974 14992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:59:31.479984 14992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:59:31.481565 14992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.193.212:38905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:37847
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:59:31.481804 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:59:31.482031 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:59:31.486886 15000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:59:31.486862 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:31.487076 14992 server_base.cc:1034] running on GCE node
W20260810 14:59:31.487188 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:31.487495 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:59:31.487777 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:59:31.488915 14992 hybrid_clock.cc:648] HybridClock initialized: now 1786373971488902 us; error 31 us; skew 500 ppm
Aug 10 14:59:31 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373971, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:59:31.491317 14992 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 14:59:31.491757 14992 webserver.cc:460] Webserver started at http://127.28.193.254:40293/ using document root <none> and password file <none>
I20260810 14:59:31.491959 14992 fs_manager.cc:359] Metadata directory not provided
I20260810 14:59:31.491997 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:59:31.492108 14992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:59:31 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373971, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:31.551102 14992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "38448d2ddff247cb8f4083831a85e4bb"
format_stamp: "Formatted at 2026-08-10 14:59:31 on dist-test-slave-cvwc"
server_key: "f3c705b3302b4d0b90bf8a07b2eab52e"
server_key_iv: "aae200236318aeabf196a22290066417"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:31.551501 14992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "38448d2ddff247cb8f4083831a85e4bb"
format_stamp: "Formatted at 2026-08-10 14:59:31 on dist-test-slave-cvwc"
server_key: "f3c705b3302b4d0b90bf8a07b2eab52e"
server_key_iv: "aae200236318aeabf196a22290066417"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:31.552953 14992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 14:59:31.567894 15019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:31.568589 14992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 14:59:31.568706 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "38448d2ddff247cb8f4083831a85e4bb"
format_stamp: "Formatted at 2026-08-10 14:59:31 on dist-test-slave-cvwc"
server_key: "f3c705b3302b4d0b90bf8a07b2eab52e"
server_key_iv: "aae200236318aeabf196a22290066417"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:31.568800 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:59:31.587301 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:59:31.589131 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:59:31.589277 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:59:31.594339 14992 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:37847
I20260810 14:59:31.594369 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:37847 every 8 connection(s)
I20260810 14:59:31.594847 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 14:59:31.595566 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:59:31.597904 15104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: Bootstrap starting.
I20260810 14:59:31.598495 15104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: Neither blocks nor log segments found. Creating new log.
I20260810 14:59:31.598788 15104 log.cc:824] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:59:31.599007 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 14992
I20260810 14:59:31.599115 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 14:59:31.599293 29447 external_mini_cluster.cc:1428] Setting key d9ed2f991a016721ba95a02d98c09f04
I20260810 14:59:31.599628 15104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: No bootstrap required, opened a new log
I20260810 14:59:31.602078 15104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } }
I20260810 14:59:31.602218 15104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:59:31.602250 15104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38448d2ddff247cb8f4083831a85e4bb, State: Initialized, Role: FOLLOWER
I20260810 14:59:31.602417 15104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } }
I20260810 14:59:31.602509 15104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:59:31.602542 15104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:59:31.602612 15104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:59:31.603462 15104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } }
I20260810 14:59:31.603562 15104 leader_election.cc:302] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38448d2ddff247cb8f4083831a85e4bb; no voters:
I20260810 14:59:31.603746 15104 leader_election.cc:288] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:59:31.603791 15110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:59:31.603997 15110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [term 1 LEADER]: Becoming Leader. State: Replica: 38448d2ddff247cb8f4083831a85e4bb, State: Running, Role: LEADER
I20260810 14:59:31.604089 15104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:59:31.604127 15110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } }
I20260810 14:59:31.604440 15110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38448d2ddff247cb8f4083831a85e4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } } }
I20260810 14:59:31.604511 15110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [sys.catalog]: This master's current role is: LEADER
I20260810 14:59:31.604624 15110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38448d2ddff247cb8f4083831a85e4bb. Latest consensus state: current_term: 1 leader_uuid: "38448d2ddff247cb8f4083831a85e4bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38448d2ddff247cb8f4083831a85e4bb" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37847 } } }
I20260810 14:59:31.604672 15110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb [sys.catalog]: This master's current role is: LEADER
I20260810 14:59:31.606024 15104 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:59:31.608779 15119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:59:31.610047 15119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 14:59:31 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:59:31.612829 15119 catalog_manager.cc:1347] Generated new cluster ID: c75522857d97450caf9829d038151dd7
I20260810 14:59:31.612874 15119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:59:31.619789 15119 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:59:31.620493 15119 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:59:31.621501 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:31.599822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.28.193.254:37847)
0810 14:59:31.600015 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:31.600026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:31.600068 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:59:31.602084 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:31.602095 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:31.602165 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:31.602329 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:59:31.603390 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:31.604855 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:31.606045 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:31.609012 (+ 2967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:31.609538 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:31.612021 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:59:31.612026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:59:31.612026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:59:31.612038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:59:31.619293 (+ 7255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:31.619721 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:31.619725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:31.619725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:31.619849 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:31.620081 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:31.620082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:31.620082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:31.620397 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:59:31.621013 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:31.621090 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 14:59:31.621258 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260810 14:59:31.623413 15119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: Generated new TSK 0
I20260810 14:59:31.623570 15119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 14:59:32.342892 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:38905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:59:32.449774 15179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:59:32.449981 15179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:59:32.450004 15179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:59:32.451197 15179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:59:32.451241 15179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:59:32.451267 15179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 14:59:32.452745 15179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.193.212:38905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:59:32.452986 15179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:59:32.453207 15179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:59:32.458226 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:59:32.458204 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:59:32.458534 15188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:32.458786 15179 server_base.cc:1034] running on GCE node
I20260810 14:59:32.458983 15179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:59:32.459208 15179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:59:32.460376 15179 hybrid_clock.cc:648] HybridClock initialized: now 1786373972460356 us; error 33 us; skew 500 ppm
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:59:32.463019 15179 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 14:59:32.463531 15179 webserver.cc:460] Webserver started at http://127.28.193.193:43047/ using document root <none> and password file <none>
I20260810 14:59:32.463896 15179 fs_manager.cc:359] Metadata directory not provided
I20260810 14:59:32.464026 15179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:59:32.464218 15179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:32.485298 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a88828167b9ff59d7a142481870b387f"
server_key_iv: "63ba448ea6e77e1ae8812f24d1504a7e"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.485842 15179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a88828167b9ff59d7a142481870b387f"
server_key_iv: "63ba448ea6e77e1ae8812f24d1504a7e"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.488387 15179 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:59:32.505283 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.505544 15179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:59:32.505635 15179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a88828167b9ff59d7a142481870b387f"
server_key_iv: "63ba448ea6e77e1ae8812f24d1504a7e"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.505759 15179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:59:32.524061 15179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:59:32.526852 15179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:59:32.527041 15179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:59:32.527374 15179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:59:32.527844 15179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:59:32.527895 15179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.527927 15179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:59:32.527985 15179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.535646 15179 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:43843
I20260810 14:59:32.535737 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:43843 every 8 connection(s)
I20260810 14:59:32.536077 15179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:59:32.542210 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 15179
I20260810 14:59:32.542316 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 14:59:32.542534 29447 external_mini_cluster.cc:1428] Setting key 82a2023c51b5dfb7503e0eabad211255
I20260810 14:59:32.546180 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:32.536887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:37847 (local address 127.28.193.193:52499)
0810 14:59:32.537091 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:32.537135 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:32.537673 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:32.538091 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:32.538096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:32.538295 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:59:32.538522 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:32.538527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.539233 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.539234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:32.540064 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:32.540066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.540433 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.540433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:32.540559 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:32.541097 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:59:32.541105 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:59:32.542501 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:59:32.544469 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:32.544477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:32.544479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:32.544776 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:32.545027 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:32.545029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:32.545029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:32.545170 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:32.545474 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:59:32.545481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:59:32.545646 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 14:59:32.545778 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 14:59:32.545943 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260810 14:59:32.546819 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:32.536933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:52499 (local address 127.28.193.254:37847)
0810 14:59:32.537820 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:32.537822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:32.537830 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:59:32.537912 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:32.537914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:32.537929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:32.537999 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:59:32.538621 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.539146 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.540156 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.540362 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.540420 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:32.542617 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:59:32.542619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:59:32.542619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:59:32.542628 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:59:32.544378 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:32.544862 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:32.544863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:32.544863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:32.544944 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:32.545234 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:32.545235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:32.545235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:32.545384 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:59:32.545445 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:32.546624 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0810 14:59:32.546719 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":87,"threads_started":1}
I20260810 14:59:32.547633 15401 heartbeater.cc:347] Connected to a master server at 127.28.193.254:37847
I20260810 14:59:32.547745 15401 heartbeater.cc:464] Registering TS with master...
I20260810 14:59:32.547950 15401 heartbeater.cc:511] Master 127.28.193.254:37847 requested a full tablet report, sending...
I20260810 14:59:32.548730 15045 ts_manager.cc:194] Registered new tserver with Master: 598ce78ec9a541e3b08b1d3bf57bf9e8 (127.28.193.193:43843)
I20260810 14:59:32.549611 15045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:52499
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 14:59:32.574618 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:38905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:59:32.693552 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:59:32.693850 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:59:32.693892 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:59:32.695508 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:59:32.695603 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:59:32.695667 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 14:59:32.698029 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.193.212:38905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:59:32.698433 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:59:32.698752 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:59:32.705132 15427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:59:32.705379 15429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:32.707413 15415 server_base.cc:1034] running on GCE node
W20260810 14:59:32.708691 15426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:32.708954 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:59:32.709218 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:59:32.710374 15415 hybrid_clock.cc:648] HybridClock initialized: now 1786373972710343 us; error 45 us; skew 500 ppm
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:59:32.713078 15415 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 14:59:32.713551 15415 webserver.cc:460] Webserver started at http://127.28.193.194:44027/ using document root <none> and password file <none>
I20260810 14:59:32.713852 15415 fs_manager.cc:359] Metadata directory not provided
I20260810 14:59:32.713917 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:59:32.714049 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:32.731542 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "e51b2d1092b742248e2f65ebe2eeb15a"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a252fb9fc81458e0d0240dae12d240eb"
server_key_iv: "54ac76b06eafb07498ae976bf9d5d8e8"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.732100 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "e51b2d1092b742248e2f65ebe2eeb15a"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a252fb9fc81458e0d0240dae12d240eb"
server_key_iv: "54ac76b06eafb07498ae976bf9d5d8e8"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.734431 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:59:32.746905 15441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.747219 15415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 14:59:32.747339 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "e51b2d1092b742248e2f65ebe2eeb15a"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "a252fb9fc81458e0d0240dae12d240eb"
server_key_iv: "54ac76b06eafb07498ae976bf9d5d8e8"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.747440 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:59:32.767462 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:59:32.769965 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:59:32.770130 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:59:32.770416 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:59:32.770820 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:59:32.770874 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.770910 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:59:32.770937 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.780117 15415 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:35247
I20260810 14:59:32.780613 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 14:59:32.781361 15626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:35247 every 8 connection(s)
I20260810 14:59:32.782217 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 15415
I20260810 14:59:32.782323 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 14:59:32.782500 29447 external_mini_cluster.cc:1428] Setting key 8878d1b5e23e72cafa0e278438f86ac1
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 14:59:32.795637 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:32.781799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:39059 (local address 127.28.193.254:37847)
0810 14:59:32.781849 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:32.781851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:32.781863 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:59:32.782517 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:32.782519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:32.782536 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:32.782571 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:59:32.783498 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.784262 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.785840 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.786164 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.786208 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:32.788773 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:59:32.788775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:59:32.788775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:59:32.788784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:59:32.790601 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:32.794380 (+ 3779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:32.794382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:32.794382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:32.794450 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:32.794746 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:32.794747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:32.794747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:32.794876 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:59:32.794952 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:32.795438 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 14:59:32.795540 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 14:59:32.795768 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:32.781438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:37847 (local address 127.28.193.194:39059)
0810 14:59:32.781606 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:32.781653 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:32.782363 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:32.782799 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:32.782804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:32.783067 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:59:32.783361 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:32.783368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.784400 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.784400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:32.785693 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:32.785697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:32.786305 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:32.786306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:32.786449 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:32.787046 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:59:32.787057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:59:32.788664 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:59:32.793911 (+ 5247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:32.793919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:32.793921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:32.794234 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:32.794521 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:32.794522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:32.794522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:32.794666 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:32.795138 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:59:32.795146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:59:32.795289 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 14:59:32.795380 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:59:32.795541 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260810 14:59:32.796260 15627 heartbeater.cc:347] Connected to a master server at 127.28.193.254:37847
I20260810 14:59:32.796361 15627 heartbeater.cc:464] Registering TS with master...
I20260810 14:59:32.796566 15627 heartbeater.cc:511] Master 127.28.193.254:37847 requested a full tablet report, sending...
I20260810 14:59:32.797004 15045 ts_manager.cc:194] Registered new tserver with Master: e51b2d1092b742248e2f65ebe2eeb15a (127.28.193.194:35247)
I20260810 14:59:32.797493 15045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:39059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 14:59:32.812927 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:38905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:59:32.924705 15638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:59:32.925077 15638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:59:32.925164 15638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:59:32.926980 15638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:59:32.927093 15638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:59:32.927181 15638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 14:59:32.929607 15638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.193.212:38905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:37847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 14:59:32.929946 15638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:59:32.930227 15638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:59:32.936743 15645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:32.936864 15638 server_base.cc:1034] running on GCE node
W20260810 14:59:32.936884 15649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:59:32.936902 15646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:59:32.937513 15638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:59:32.937958 15638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:59:32.939152 15638 hybrid_clock.cc:648] HybridClock initialized: now 1786373972939125 us; error 48 us; skew 500 ppm
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:59:32.942670 15638 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 14:59:32.943241 15638 webserver.cc:460] Webserver started at http://127.28.193.195:39343/ using document root <none> and password file <none>
I20260810 14:59:32.943503 15638 fs_manager.cc:359] Metadata directory not provided
I20260810 14:59:32.943565 15638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:59:32.943706 15638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:59:32 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:32.960753 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "612ae3cc8d264bec929cd04b5e93cab5"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "4f042d2c6f23cd69d9812a58ef57453e"
server_key_iv: "3cef772e33664ec2e119e0bf6c793e94"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.961205 15638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "612ae3cc8d264bec929cd04b5e93cab5"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "4f042d2c6f23cd69d9812a58ef57453e"
server_key_iv: "3cef772e33664ec2e119e0bf6c793e94"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.966174 15638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 14:59:32.979346 15658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.979563 15638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:59:32.979702 15638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "612ae3cc8d264bec929cd04b5e93cab5"
format_stamp: "Formatted at 2026-08-10 14:59:32 on dist-test-slave-cvwc"
server_key: "4f042d2c6f23cd69d9812a58ef57453e"
server_key_iv: "3cef772e33664ec2e119e0bf6c793e94"
server_key_version: "kuduclusterkey@0"
I20260810 14:59:32.979808 15638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:59:32.995810 15638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:59:32.998080 15638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:59:32.998212 15638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:59:32.998457 15638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:59:32.998847 15638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:59:32.998886 15638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:32.998917 15638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:59:32.998934 15638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:59:33.008462 15638 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:33973
I20260810 14:59:33.011214 15850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:33973 every 8 connection(s)
I20260810 14:59:33.011586 15638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 14:59:33 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 14:59:33.020689 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 15638
I20260810 14:59:33.020812 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 14:59:33.020996 29447 external_mini_cluster.cc:1428] Setting key 652e07064509e743f3ab0072c57d6f14
I20260810 14:59:33.024428 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.012923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:51599 (local address 127.28.193.254:37847)
0810 14:59:33.012968 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.012970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.013173 (+ 203us) server_negotiation.cc:407] Connection header received
0810 14:59:33.013911 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.013913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.013927 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.013957 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:59:33.014811 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.015627 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.017105 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.017409 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.017457 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.020130 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:59:33.020132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:59:33.020132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:59:33.020140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:59:33.021879 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:33.023263 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:33.023265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:33.023265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:33.023337 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:33.023638 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:33.023639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:33.023640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:33.023803 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:59:33.023849 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.024235 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.024329 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 14:59:33.024982 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.012615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:37847 (local address 127.28.193.195:51599)
0810 14:59:33.013042 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.013088 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.013785 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.014157 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.014161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.014423 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:59:33.014698 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.014704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.015743 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.015744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.017002 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.017005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.017558 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.017559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.017749 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.018313 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:59:33.018323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:59:33.020011 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:59:33.022640 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:33.022647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:33.022649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:33.023155 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:33.023411 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:59:33.023412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:59:33.023412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:59:33.023560 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:59:33.023898 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:59:33.023905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:59:33.024083 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.024562 (+ 479us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.024730 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":99,"threads_started":1}
I20260810 14:59:33.025411 15851 heartbeater.cc:347] Connected to a master server at 127.28.193.254:37847
I20260810 14:59:33.025513 15851 heartbeater.cc:464] Registering TS with master...
I20260810 14:59:33.025699 15851 heartbeater.cc:511] Master 127.28.193.254:37847 requested a full tablet report, sending...
I20260810 14:59:33.026077 15045 ts_manager.cc:194] Registered new tserver with Master: 612ae3cc8d264bec929cd04b5e93cab5 (127.28.193.195:33973)
I20260810 14:59:33.026562 15045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:51599
I20260810 14:59:33.033247 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:59:33.043150 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.035000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.28.193.254:37847)
0810 14:59:33.035042 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.035044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.035390 (+ 346us) server_negotiation.cc:407] Connection header received
0810 14:59:33.035487 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.035488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.035499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.035529 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:59:33.036334 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.037067 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.038187 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.038448 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.038561 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.040043 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:59:33.040045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:59:33.040045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:59:33.040052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:59:33.041818 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:33.042241 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:33.042242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:33.042243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:33.042302 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:59:33.042531 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:59:33.042532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:59:33.042532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:59:33.042662 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:59:33.042772 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.042966 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.043048 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 14:59:33.045457 15045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:59:33.048157 15045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:59:33.057673 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.052054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.28.193.195:33973)
0810 14:59:33.052481 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.052486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.053037 (+ 551us) server_negotiation.cc:407] Connection header received
0810 14:59:33.053144 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.053147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.053163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.053215 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:59:33.053432 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.054624 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.056340 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.057136 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.057205 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.057263 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.057533 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.057577 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":140,"threads_started":1}
I20260810 14:59:33.057889 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.051861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:35247 (local address 127.0.0.1:48306)
0810 14:59:33.052710 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.052754 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.052866 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.053197 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.053201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.053204 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:59:33.053350 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.053354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.055087 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.055088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.056194 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.056198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.057596 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.057597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.057663 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.057668 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.057702 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.057804 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":1,"thread_start_us":772,"threads_started":1}
I20260810 14:59:33.058228 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.051865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:33973 (local address 127.0.0.1:57702)
0810 14:59:33.052976 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.052985 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.053063 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.053304 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.053305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.053307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:59:33.053360 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.053362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.054726 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.054727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.056213 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.056217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.057327 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.057328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.057416 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.057430 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.057834 (+ 404us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.057883 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":4,"thread_start_us":467,"threads_started":1}
I20260810 14:59:33.058775 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.052090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.28.193.194:35247)
0810 14:59:33.052976 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.052981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.052995 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:59:33.053044 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.053046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.053065 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.053444 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:59:33.053532 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.054973 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.056342 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.057478 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.058516 (+ 1038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.058595 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.058628 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.058675 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":97,"threads_started":1}
I20260810 14:59:33.059119 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 06d8645944724388bc5335a8ad83b5e6 (DEFAULT_TABLE table=test-table [id=51a65e85770144bbb39c077966e9bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:59:33.059520 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d8645944724388bc5335a8ad83b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:59:33.059607 15518 tablet_service.cc:1467] Processing CreateTablet for tablet 06d8645944724388bc5335a8ad83b5e6 (DEFAULT_TABLE table=test-table [id=51a65e85770144bbb39c077966e9bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:59:33.060019 15518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d8645944724388bc5335a8ad83b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:59:33.064399 15878 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Bootstrap starting.
I20260810 14:59:33.065421 15877 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Bootstrap starting.
I20260810 14:59:33.066133 15878 tablet_bootstrap.cc:654] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Neither blocks nor log segments found. Creating new log.
I20260810 14:59:33.066577 15877 tablet_bootstrap.cc:654] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Neither blocks nor log segments found. Creating new log.
I20260810 14:59:33.066569 15878 log.cc:824] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:59:33.066915 15877 log.cc:824] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:59:33.067692 15878 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: No bootstrap required, opened a new log
I20260810 14:59:33.067790 15878 ts_tablet_manager.cc:1397] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:59:33.069754 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.052775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43843 (local address 127.0.0.1:36694)
0810 14:59:33.057981 (+ 5206us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.057992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.058034 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.063895 (+ 5861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.063896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.063899 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:59:33.063945 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.063947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.067284 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.067285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.068468 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.068472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.069479 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.069480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.069547 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.069552 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.069585 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.069635 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5194,"thread_start_us":85,"threads_started":1}
I20260810 14:59:33.069924 15878 raft_consensus.cc:348] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.070104 15878 raft_consensus.cc:374] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:59:33.070142 15878 raft_consensus.cc:729] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51b2d1092b742248e2f65ebe2eeb15a, State: Initialized, Role: FOLLOWER
I20260810 14:59:33.070169 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.052885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.28.193.193:43843)
0810 14:59:33.063701 (+ 10816us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.063706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.063718 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:59:33.063756 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.063758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.063775 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.064033 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:59:33.064117 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.065209 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.068583 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.069357 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.069943 (+ 586us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.070022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.070051 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.070095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10774,"thread_start_us":58,"threads_started":1}
I20260810 14:59:33.070264 15878 consensus_queue.cc:260] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.070601 15878 ts_tablet_manager.cc:1428] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:59:33.070842 15289 tablet_service.cc:1467] Processing CreateTablet for tablet 06d8645944724388bc5335a8ad83b5e6 (DEFAULT_TABLE table=test-table [id=51a65e85770144bbb39c077966e9bd39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:59:33.071030 15877 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: No bootstrap required, opened a new log
I20260810 14:59:33.071092 15877 ts_tablet_manager.cc:1397] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 14:59:33.071223 15289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06d8645944724388bc5335a8ad83b5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:59:33.073182 15877 raft_consensus.cc:348] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.073351 15877 raft_consensus.cc:374] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:59:33.073410 15877 raft_consensus.cc:729] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 612ae3cc8d264bec929cd04b5e93cab5, State: Initialized, Role: FOLLOWER
I20260810 14:59:33.073531 15877 consensus_queue.cc:260] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.073889 15851 heartbeater.cc:503] Master 127.28.193.254:37847 was elected leader, sending a full tablet report...
I20260810 14:59:33.073820 15877 ts_tablet_manager.cc:1428] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 14:59:33.074473 15627 heartbeater.cc:503] Master 127.28.193.254:37847 was elected leader, sending a full tablet report...
I20260810 14:59:33.074601 15883 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: Bootstrap starting.
I20260810 14:59:33.075556 15883 tablet_bootstrap.cc:654] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: Neither blocks nor log segments found. Creating new log.
I20260810 14:59:33.075872 15883 log.cc:824] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:59:33.076673 15883 tablet_bootstrap.cc:492] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: No bootstrap required, opened a new log
I20260810 14:59:33.076732 15883 ts_tablet_manager.cc:1397] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:59:33.079375 15883 raft_consensus.cc:348] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.079551 15883 raft_consensus.cc:374] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:59:33.079591 15883 raft_consensus.cc:729] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598ce78ec9a541e3b08b1d3bf57bf9e8, State: Initialized, Role: FOLLOWER
I20260810 14:59:33.079728 15883 consensus_queue.cc:260] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.080073 15883 ts_tablet_manager.cc:1428] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 14:59:33.080207 15401 heartbeater.cc:503] Master 127.28.193.254:37847 was elected leader, sending a full tablet report...
Aug 10 14:59:33 dist-test-slave-cvwc krb5kdc[14257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373972, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 14:59:33.178498 15882 raft_consensus.cc:482] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:59:33.178653 15882 raft_consensus.cc:504] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.179106 15882 leader_election.cc:288] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e51b2d1092b742248e2f65ebe2eeb15a (127.28.193.194:35247), 598ce78ec9a541e3b08b1d3bf57bf9e8 (127.28.193.193:43843)
I20260810 14:59:33.183458 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.179426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43843 (local address 127.28.193.195:44259)
0810 14:59:33.179683 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.179694 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.179742 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.180116 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.180117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.180119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:59:33.180189 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.180191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.181102 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.181103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.182355 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.182359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.183199 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.183200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.183279 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.183283 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.183326 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.183362 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":145,"threads_started":1}
I20260810 14:59:33.183749 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.179917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44259 (local address 127.28.193.193:43843)
0810 14:59:33.179970 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.179972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.179980 (+ 8us) server_negotiation.cc:407] Connection header received
0810 14:59:33.180009 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.180010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.180019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.180067 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:59:33.180254 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.181006 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.182489 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.183084 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.183153 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.183163 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.183640 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.183684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 14:59:33.184018 15322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d8645944724388bc5335a8ad83b5e6" candidate_uuid: "612ae3cc8d264bec929cd04b5e93cab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" is_pre_election: true
I20260810 14:59:33.184188 15322 raft_consensus.cc:2454] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 612ae3cc8d264bec929cd04b5e93cab5 in term 0.
I20260810 14:59:33.184491 15662 leader_election.cc:302] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598ce78ec9a541e3b08b1d3bf57bf9e8, 612ae3cc8d264bec929cd04b5e93cab5; no voters:
I20260810 14:59:33.184633 15882 raft_consensus.cc:2781] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:59:33.184685 15882 raft_consensus.cc:482] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:59:33.184729 15882 raft_consensus.cc:3037] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:59:33.185643 15882 raft_consensus.cc:504] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.185815 15882 leader_election.cc:288] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [CANDIDATE]: Term 1 election: Requested vote from peers e51b2d1092b742248e2f65ebe2eeb15a (127.28.193.194:35247), 598ce78ec9a541e3b08b1d3bf57bf9e8 (127.28.193.193:43843)
I20260810 14:59:33.186034 15322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d8645944724388bc5335a8ad83b5e6" candidate_uuid: "612ae3cc8d264bec929cd04b5e93cab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8"
I20260810 14:59:33.186117 15322 raft_consensus.cc:3037] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:59:33.186563 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.179893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:40525 (local address 127.28.193.194:35247)
0810 14:59:33.179939 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.179941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.179950 (+ 9us) server_negotiation.cc:407] Connection header received
0810 14:59:33.179980 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.179981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.179990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.180040 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:59:33.183699 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.184348 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.185635 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.186153 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.186203 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.186213 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.186456 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.186497 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 14:59:33.186836 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.179336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:35247 (local address 127.28.193.195:40525)
0810 14:59:33.179675 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.179690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:59:33.179761 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:59:33.183509 (+ 3748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:59:33.183510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:59:33.183514 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:59:33.183605 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.183607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.184473 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.184474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.185534 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:59:33.185537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.186273 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.186274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:59:33.186371 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.186375 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:59:33.186683 (+ 308us) client_negotiation.cc:240] Negotiation successful
0810 14:59:33.186724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260810 14:59:33.186964 15322 raft_consensus.cc:2454] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 612ae3cc8d264bec929cd04b5e93cab5 in term 1.
I20260810 14:59:33.187249 15552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d8645944724388bc5335a8ad83b5e6" candidate_uuid: "612ae3cc8d264bec929cd04b5e93cab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" is_pre_election: true
I20260810 14:59:33.187417 15662 leader_election.cc:302] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598ce78ec9a541e3b08b1d3bf57bf9e8, 612ae3cc8d264bec929cd04b5e93cab5; no voters:
I20260810 14:59:33.187464 15552 raft_consensus.cc:2454] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 612ae3cc8d264bec929cd04b5e93cab5 in term 0.
I20260810 14:59:33.187512 15882 raft_consensus.cc:2781] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:59:33.187705 15882 raft_consensus.cc:686] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 1 LEADER]: Becoming Leader. State: Replica: 612ae3cc8d264bec929cd04b5e93cab5, State: Running, Role: LEADER
I20260810 14:59:33.187808 15882 consensus_queue.cc:237] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } }
I20260810 14:59:33.189184 15045 catalog_manager.cc:5495] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 612ae3cc8d264bec929cd04b5e93cab5 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "612ae3cc8d264bec929cd04b5e93cab5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "612ae3cc8d264bec929cd04b5e93cab5" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 33973 } health_report { overall_health: HEALTHY } } }
I20260810 14:59:33.187247 15550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06d8645944724388bc5335a8ad83b5e6" candidate_uuid: "612ae3cc8d264bec929cd04b5e93cab5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b2d1092b742248e2f65ebe2eeb15a"
I20260810 14:59:33.189617 15550 raft_consensus.cc:3037] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:59:33.190513 15550 raft_consensus.cc:2454] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 612ae3cc8d264bec929cd04b5e93cab5 in term 1.
I20260810 14:59:33.205655 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.202546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.28.193.195:33973)
0810 14:59:33.202597 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 14:59:33.202599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 14:59:33.202606 (+ 7us) server_negotiation.cc:407] Connection header received
0810 14:59:33.202638 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:59:33.202639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:59:33.202655 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:59:33.202711 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:59:33.202875 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.203649 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.204499 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:59:33.204777 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:59:33.204839 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:59:33.205081 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:59:33.205219 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:59:33.205304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:59:33.205554 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:59:33.205577 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:59:33.208081 15322 raft_consensus.cc:1261] T 06d8645944724388bc5335a8ad83b5e6 P 598ce78ec9a541e3b08b1d3bf57bf9e8 [term 1 FOLLOWER]: Refusing update from remote peer 612ae3cc8d264bec929cd04b5e93cab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:59:33.208467 15882 consensus_queue.cc:1035] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598ce78ec9a541e3b08b1d3bf57bf9e8" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:59:33.209040 15550 raft_consensus.cc:1261] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 1 FOLLOWER]: Refusing update from remote peer 612ae3cc8d264bec929cd04b5e93cab5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:59:33.210495 15890 consensus_queue.cc:1035] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:59:33.213414 15895 mvcc.cc:205] Tried to move back new op lower bound from 7316987794255921152 to 7316987794177986560. Current Snapshot: MvccSnapshot[applied={T|T < 7316987794255921152}]
I20260810 14:59:33.215497 15894 mvcc.cc:205] Tried to move back new op lower bound from 7316987794255921152 to 7316987794177986560. Current Snapshot: MvccSnapshot[applied={T|T < 7316987794255921152}]
I20260810 14:59:33.218102 15045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:59:33.218211 15045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:59:33.219031 15045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38448d2ddff247cb8f4083831a85e4bb: Sending DeleteTablet for 3 replicas of tablet 06d8645944724388bc5335a8ad83b5e6
I20260810 14:59:33.219571 15518 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d8645944724388bc5335a8ad83b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:59:33 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:48306
I20260810 14:59:33.219825 15903 tablet_replica.cc:330] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: stopping tablet replica
I20260810 14:59:33.219807 15289 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d8645944724388bc5335a8ad83b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:59:33 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:36694
I20260810 14:59:33.220026 15903 raft_consensus.cc:2229] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:59:33.220057 15739 tablet_service.cc:1514] Processing DeleteTablet for tablet 06d8645944724388bc5335a8ad83b5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:59:33 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:57702
I20260810 14:59:33.220191 15903 pending_rounds.cc:62] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Trying to abort 2 pending ops.
I20260810 14:59:33.220260 15903 pending_rounds.cc:68] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316987794255921152 op_type: WRITE_OP write_request { tablet_id: "06d8645944724388bc5335a8ad83b5e6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\201\321\347\323\006\"8\n\ntest-admin\022*\n 51a65e85770144bbb39c077966e9bd39\020\001\030\001 \001(\001" signature: "<redacted>""Gc\231\350i\037\"\3462\204a\337\347tj\272\307\276\033\241<\254\253>&4\030\320\357\244\253\336V\032\274\303z\251C\033\220h-P`\364\004\0059:[\223J\246\317\324\005\205\032\341~b[S" signing_key_seq_num: 0 } } request_id { client_id: "ab6782c49f4a413990722e8c0a18496e" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260810 14:59:33.220613 15903 pending_rounds.cc:68] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316987794177986560 op_type: NO_OP noop_request { }
I20260810 14:59:33.220659 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 15179
I20260810 14:59:33.220749 15903 raft_consensus.cc:2866] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 14:59:33.220821 15903 raft_consensus.cc:2258] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:59:33.233860 15910 tablet_replica.cc:330] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: stopping tablet replica
I20260810 14:59:33.234017 15910 raft_consensus.cc:2229] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:59:33.234144 15910 raft_consensus.cc:2258] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:59:33.234680 15910 ts_tablet_manager.cc:1905] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 14:59:33.242266 15898 consensus_queue.cc:1159] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e51b2d1092b742248e2f65ebe2eeb15a" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260810 14:59:33.242498 15910 ts_tablet_manager.cc:1918] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:59:33.242586 15910 log.cc:1196] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/06d8645944724388bc5335a8ad83b5e6
I20260810 14:59:33.242873 15910 ts_tablet_manager.cc:1939] T 06d8645944724388bc5335a8ad83b5e6 P 612ae3cc8d264bec929cd04b5e93cab5: Deleting consensus metadata
I20260810 14:59:33.243387 15023 catalog_manager.cc:4841] TS 612ae3cc8d264bec929cd04b5e93cab5 (127.28.193.195:33973): tablet 06d8645944724388bc5335a8ad83b5e6 (table test-table [id=51a65e85770144bbb39c077966e9bd39]) successfully deleted
I20260810 14:59:33.249537 15903 ts_tablet_manager.cc:1905] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:59:33.251157 15903 ts_tablet_manager.cc:1918] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:59:33.251226 15903 log.cc:1196] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/06d8645944724388bc5335a8ad83b5e6
I20260810 14:59:33.251458 15903 ts_tablet_manager.cc:1939] T 06d8645944724388bc5335a8ad83b5e6 P e51b2d1092b742248e2f65ebe2eeb15a: Deleting consensus metadata
I20260810 14:59:33.252017 15027 catalog_manager.cc:4841] TS e51b2d1092b742248e2f65ebe2eeb15a (127.28.193.194:35247): tablet 06d8645944724388bc5335a8ad83b5e6 (table test-table [id=51a65e85770144bbb39c077966e9bd39]) successfully deleted
W20260810 14:59:33.253219 15027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:43843 (error 108)
I20260810 14:59:33.262203 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.262042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43843 (local address 127.0.0.1:36710)
0810 14:59:33.262101 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.262155 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 14:59:33.262364 15027 catalog_manager.cc:4584] TS 598ce78ec9a541e3b08b1d3bf57bf9e8 (127.28.193.193:43843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 06d8645944724388bc5335a8ad83b5e6: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43843: connect: Connection refused (error 111)
I20260810 14:59:33.262530 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 15415
I20260810 14:59:33.311630 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 15638
I20260810 14:59:33.317763 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.317552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43843 (local address 127.0.0.1:36720)
0810 14:59:33.317612 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.317663 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 14:59:33.318115 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:59:33.318044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43843 (local address 127.0.0.1:36730)
0810 14:59:33.318074 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 14:59:33.318095 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260810 14:59:33.339833 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 14992
I20260810 14:59:33.358425 29447 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:59:33.706637 29447 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:59:33.706746 29447 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:59:34.051091 29447 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:59:34.051 UTC [14299] LOG: received smart shutdown request
2026-08-10 14:59:34.055 UTC [14299] LOG: background worker "logical replication launcher" (PID 14315) exited with exit code 1
2026-08-10 14:59:34.055 UTC [14307] LOG: shutting down
2026-08-10 14:59:34.191 UTC [14299] LOG: database system is shut down
2026-08-10T14:59:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46786 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:59:34 dist-test-slave-cvwc krb5kdc[15925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:59:34 dist-test-slave-cvwc krb5kdc[15925](info): set up 2 sockets
Aug 10 14:59:34 dist-test-slave-cvwc krb5kdc[15925](info): commencing operation
krb5kdc: starting...
W20260810 14:59:35.424721 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 14:59:35 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786373975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:59:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangeradmin_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangeradmin_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangerlookup_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangerlookup_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
I20260810 14:59:35.579576 29447 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskILwtmC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/postgres -l logfile start
2026-08-10 14:59:37.824 UTC [15977] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:59:37.824 UTC [15977] LOG: listening on IPv4 address "127.28.193.212", port 57389
2026-08-10 14:59:37.827 UTC [15977] LOG: listening on Unix socket "/tmp/.s.PGSQL.57389"
2026-08-10 14:59:37.847 UTC [15981] LOG: database system was shut down at 2026-08-10 14:59:36 UTC
2026-08-10 14:59:37.850 UTC [15977] LOG: database system is ready to accept connections
I20260810 14:59:38.951400 29447 mini_postgres.cc:96] Postgres bound to 57389
2026-08-10 14:59:38.958 UTC [15993] FATAL: database "slave" does not exist
127.28.193.212:57389 - accepting connections
I20260810 14:59:38.958500 29447 mini_ranger.cc:161] Starting Ranger...
I20260810 14:59:38.972726 29447 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:59:39.388413 29447 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:59:39.388486 29447 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin
2026-08-10 14:59:39,667 [I] DB FLAVOR :POSTGRES
2026-08-10 14:59:39,667 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:59:39,667 [I] Checking connection..
2026-08-10 14:59:39,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:59:39,923 [I] Checking connection passed.
2026-08-10 14:59:39,923 [I] --------- Verifying version history table ---------
2026-08-10 14:59:39,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:59:40,184 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:59:40,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:59:40,419 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:59:40,419 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:59:40,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:59:40.645 UTC [16096] WARNING: there is no transaction in progress
2026-08-10 14:59:40,652 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:59:40,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:59:40,897 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:59:40,897 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:59:40,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:59:41,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:59:41,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cvwc.c.gcp-upstream.internal','N') ;"
2026-08-10 14:59:41,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:59:41,920 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:59:41,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:59:42,180 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:59:42,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:59:42,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:59:42,676 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:59:42,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:59:43.594 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.609 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.625 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.641 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.656 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.672 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.733 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.740 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.753 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.763 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.773 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.786 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.795 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.801 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.807 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:43.812 UTC [16271] WARNING: there is no transaction in progress
2026-08-10 14:59:44,380 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:59:44,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:59:44,621 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:59:44,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:59:44,879 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:59:44,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:59:45,135 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:59:45,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:59:45,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:59:45,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cvwc.c.gcp-upstream.internal';"
2026-08-10 14:59:45,876 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:59:45,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:59:46,132 [I] DB_PATCHES have already been applied
I20260810 14:59:46.137835 29447 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskILwtmC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:59:46.137917 29447 mini_ranger.cc:193] Using host: 127.28.193.212
I20260810 14:59:46.141495 29447 mini_ranger.cc:227] Ranger admin URL: http://127.28.193.212:35407
Aug 10, 2026 2:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:59:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:59:47.346779 29447 mini_ranger.cc:160] Time spent starting Ranger: real 8.388s user 0.001s sys 0.008s
Aug 10, 2026 3:00:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:00:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 3:00:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:00:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:00:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 15:00:07.153400 29447 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.193.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.193.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangerkms_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.193.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/rangerkms_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.193.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.193.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.193.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/HTTP_127.28.193.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 15:00:07 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374007, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 15:00:07.238869 29447 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 15:00:07.248523 29447 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 15:00:07.991405 29447 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 15:00:07.991489 29447 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms
I20260810 15:00:07.991505 29447 mini_ranger_kms.cc:227] Using postgres at 127.28.193.212:57389
2026-08-10 15:00:08,147 [I] DB FLAVOR :POSTGRES
2026-08-10 15:00:08,148 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:00:08,148 [I] Checking connection
2026-08-10 15:00:08,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 15:00:08,401 [I] connection success
2026-08-10 15:00:08,401 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 15:00:08,401 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 15:00:08,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 15:00:08,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 15:00:08,642 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:00:08,643 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 15:00:08,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.193.212:57389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 15:00:08,920 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 15:00:09.373627 29447 mini_ranger_kms.cc:310] Created kms service
I20260810 15:00:09.582120 29447 mini_ranger_kms.cc:326] Created kudu user
I20260810 15:00:09.629365 29447 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 15:00:09 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374007, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:10.070982 29447 mini_ranger_kms.cc:384] Added ranger policy
I20260810 15:00:10.071216 29447 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms:/tmp/dist-test-taskILwtmC/build/release/bin/postgresql.jar:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskILwtmC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 15:00:10.071244 29447 mini_ranger_kms.cc:240] Using host: 127.28.193.212
I20260810 15:00:10.075736 29447 mini_ranger_kms.cc:276] Ranger KMS PID: 16587
I20260810 15:00:10.075801 29447 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.193.212:51427
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:00:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 15:00:12 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786374012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:12.659092 29447 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.420s user 0.004s sys 0.137s
I20260810 15:00:12.659202 29447 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 15:00:12.659245 29447 mini_ranger_kms.cc:401] 127.28.193.212:51427/kms/v1/keys
Aug 10 15:00:12 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
Aug 10 15:00:13 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374012, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.193.212@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
Aug 10 15:00:15 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:00:15.521207 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:38023
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:38023
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:15.617527 16675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:15.617765 16675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:15.617800 16675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:15.619020 16675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:15.619064 16675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:15.619082 16675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:15.619098 16675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:15.619113 16675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:15.620714 16675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.193.212:51427/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:38023
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:15.620962 16675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:15.621170 16675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:15.626050 16685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:15.626058 16683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:15.626133 16690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:15.626439 16675 server_base.cc:1034] running on GCE node
I20260810 15:00:15.626695 16675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:15.626994 16675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:15.628142 16675 hybrid_clock.cc:648] HybridClock initialized: now 1786374015628113 us; error 43 us; skew 500 ppm
Aug 10 15:00:15 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:15.631279 16675 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:15.631704 16675 webserver.cc:460] Webserver started at http://127.28.193.254:37945/ using document root <none> and password file <none>
I20260810 15:00:15.631912 16675 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:15.631954 16675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:15.632094 16675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:00:15 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374015, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:15.679889 16675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "79776cb8c32344c28495b9d469922de6"
format_stamp: "Formatted at 2026-08-10 15:00:15 on dist-test-slave-cvwc"
server_key: "d2f8b881a35dc0df23ba93e4023a5fb3"
server_key_iv: "14db4b80b4fd65bba1fa6b61ec803eda"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:15.680320 16675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "79776cb8c32344c28495b9d469922de6"
format_stamp: "Formatted at 2026-08-10 15:00:15 on dist-test-slave-cvwc"
server_key: "d2f8b881a35dc0df23ba93e4023a5fb3"
server_key_iv: "14db4b80b4fd65bba1fa6b61ec803eda"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:15.681811 16675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 15:00:15.696499 16702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:15.697856 16675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:15.697970 16675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "79776cb8c32344c28495b9d469922de6"
format_stamp: "Formatted at 2026-08-10 15:00:15 on dist-test-slave-cvwc"
server_key: "d2f8b881a35dc0df23ba93e4023a5fb3"
server_key_iv: "14db4b80b4fd65bba1fa6b61ec803eda"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:15.698066 16675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:15.714174 16675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:15.716092 16675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:15.716229 16675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:15.721010 16675 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:38023
I20260810 15:00:15.721076 16796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:38023 every 8 connection(s)
I20260810 15:00:15.721444 16675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:15.722309 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:15.725605 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Bootstrap starting.
I20260810 15:00:15.726504 16799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:15.726835 16799 log.cc:824] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:15.727623 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: No bootstrap required, opened a new log
I20260810 15:00:15.728267 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 16675
I20260810 15:00:15.728371 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:00:15.728541 29447 external_mini_cluster.cc:1428] Setting key f8d292ab8977eaf50990b9ce28107599
I20260810 15:00:15.730515 16799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:15.730734 16799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:15.730805 16799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79776cb8c32344c28495b9d469922de6, State: Initialized, Role: FOLLOWER
I20260810 15:00:15.730953 16799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:15.731031 16799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:15.731060 16799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:15.731120 16799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:15.731810 16799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:15.731922 16799 leader_election.cc:302] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79776cb8c32344c28495b9d469922de6; no voters:
I20260810 15:00:15.732123 16799 leader_election.cc:288] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:00:15.732169 16808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:15.732391 16808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 LEADER]: Becoming Leader. State: Replica: 79776cb8c32344c28495b9d469922de6, State: Running, Role: LEADER
I20260810 15:00:15.732470 16799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:15.732582 16808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:15.732985 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79776cb8c32344c28495b9d469922de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } } }
I20260810 15:00:15.733124 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:15.733294 16808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79776cb8c32344c28495b9d469922de6. Latest consensus state: current_term: 1 leader_uuid: "79776cb8c32344c28495b9d469922de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } } }
I20260810 15:00:15.733388 16808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:15.733561 16824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:15.733950 16799 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:00:15.734177 16824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 15:00:15 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:15.743620 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:15.729040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.28.193.254:38023)
0810 15:00:15.729212 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:15.729222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:15.729254 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:00:15.730476 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:15.730485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:15.730538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:15.730669 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:15.731443 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:15.732624 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:15.733669 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:15.734405 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:15.734801 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:15.737585 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:15.737630 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:15.737630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:15.737674 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:15.741435 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:15.741877 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:15.741880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:15.741880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:15.742003 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:15.742277 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:15.742278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:15.742278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:15.742637 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:15.743134 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:15.743210 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:00:15.743375 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260810 15:00:15.746556 16843 mvcc.cc:205] Tried to move back new op lower bound from 7316987968495665152 to 7316987968441188352. Current Snapshot: MvccSnapshot[applied={T|T < 7316987968495665152}]
I20260810 15:00:15.749487 16824 catalog_manager.cc:1347] Generated new cluster ID: ea8e7dd7685a414ca501546d20cca252
I20260810 15:00:15.749547 16824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:00:15.774394 16824 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:00:15.775270 16824 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:15.791980 16824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Generated new TSK 0
I20260810 15:00:15.792299 16824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:00:16.478123 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:16.563625 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:16.563822 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:16.563843 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:16.564960 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:16.565002 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:16.565026 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:16.566510 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:16.566721 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:16.566938 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:16.572777 16888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:16.572813 16889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:16.572961 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:16.572871 16880 server_base.cc:1034] running on GCE node
I20260810 15:00:16.573166 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:16.573365 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:16.574517 16880 hybrid_clock.cc:648] HybridClock initialized: now 1786374016574492 us; error 37 us; skew 500 ppm
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:16.577772 16880 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:16.578310 16880 webserver.cc:460] Webserver started at http://127.28.193.193:34815/ using document root <none> and password file <none>
I20260810 15:00:16.578603 16880 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:16.578671 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:16.578855 16880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:16.595573 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "486762e5a42b446fbb3959a310cd9c70"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "b10d3aba5e5fc90e57c0c99359714b8a"
server_key_iv: "593b079c111ee6819bc68406f80ca57b"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.596130 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "486762e5a42b446fbb3959a310cd9c70"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "b10d3aba5e5fc90e57c0c99359714b8a"
server_key_iv: "593b079c111ee6819bc68406f80ca57b"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.598889 16880 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:16.613726 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.613996 16880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:16.614094 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "486762e5a42b446fbb3959a310cd9c70"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "b10d3aba5e5fc90e57c0c99359714b8a"
server_key_iv: "593b079c111ee6819bc68406f80ca57b"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.614185 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:16.637883 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:16.639868 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:16.640090 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:16.640419 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:16.640877 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:16.640980 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.641077 16880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:16.641151 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.648121 16880 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:34141
I20260810 15:00:16.648655 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:00:16.653512 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 16880
I20260810 15:00:16.653607 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:00:16.653815 29447 external_mini_cluster.cc:1428] Setting key 9b2710907475e3247deae3b9735b61a0
I20260810 15:00:16.650595 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:34141 every 8 connection(s)
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:16.664194 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:16.651524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.193:36741)
0810 15:00:16.651691 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:16.651736 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:16.652253 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:16.652516 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:16.652520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:16.652724 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:16.652920 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:16.652924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.653864 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.653866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:16.657395 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:16.657400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.657948 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.657950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:16.658066 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:16.658688 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:16.658699 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:16.660397 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:16.662441 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:16.662450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:16.662453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:16.662757 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:16.662990 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:16.662990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:16.662991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:16.663130 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:16.663467 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:16.663474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:16.663652 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 15:00:16.663761 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:00:16.663927 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 15:00:16.664778 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:16.651561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:36741 (local address 127.28.193.254:38023)
0810 15:00:16.651782 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:16.651784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:16.651837 (+ 53us) server_negotiation.cc:407] Connection header received
0810 15:00:16.652360 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:16.652361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:16.652378 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:16.652445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:16.653012 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.653688 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.657521 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.657841 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.657907 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:16.660516 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:16.660517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:16.660518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:16.660527 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:16.662340 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:16.662848 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:16.662849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:16.662850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:16.662922 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:16.663204 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:16.663205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:16.663205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:16.663371 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:16.663428 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:16.664586 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0810 15:00:16.664680 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260810 15:00:16.665365 17098 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:16.665489 17098 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:16.665686 17098 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:16.666335 16730 ts_manager.cc:194] Registered new tserver with Master: 486762e5a42b446fbb3959a310cd9c70 (127.28.193.193:34141)
I20260810 15:00:16.666952 16730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:36741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:00:16.685168 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:16.804697 17108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:16.804962 17108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:16.805002 17108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:16.806654 17108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:16.806713 17108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:16.806752 17108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:16.808874 17108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:16.809135 17108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:16.809394 17108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:16.815230 17121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:16.815377 17118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:16.815230 17117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:16.815618 17108 server_base.cc:1034] running on GCE node
I20260810 15:00:16.815802 17108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:16.816030 17108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:16.817215 17108 hybrid_clock.cc:648] HybridClock initialized: now 1786374016817191 us; error 34 us; skew 500 ppm
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:16.819976 17108 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:16.820442 17108 webserver.cc:460] Webserver started at http://127.28.193.194:35863/ using document root <none> and password file <none>
I20260810 15:00:16.820673 17108 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:16.820719 17108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:16.820832 17108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:16.843801 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "75f1b599b38246569fd29f0a7072019f"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "85fc71252a8e36b02eedd771f64f3ee3"
server_key_iv: "8609234f0a009dc9660c66ad0df28ac9"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.851305 17108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "75f1b599b38246569fd29f0a7072019f"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "85fc71252a8e36b02eedd771f64f3ee3"
server_key_iv: "8609234f0a009dc9660c66ad0df28ac9"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.854055 17108 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:16.873881 17131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.874166 17108 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 15:00:16.874285 17108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "75f1b599b38246569fd29f0a7072019f"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "85fc71252a8e36b02eedd771f64f3ee3"
server_key_iv: "8609234f0a009dc9660c66ad0df28ac9"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:16.874389 17108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:16.902916 17108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:16.905404 17108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:16.905576 17108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:16.905848 17108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:16.906262 17108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:16.906315 17108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.906351 17108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:16.906428 17108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:16.914726 17108 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44461
I20260810 15:00:16.915266 17108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:00:16.915508 17328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44461 every 8 connection(s)
I20260810 15:00:16.917611 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 17108
I20260810 15:00:16.917749 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:00:16.917935 29447 external_mini_cluster.cc:1428] Setting key afd65b0f00a41c9a04c7fd5bdc6514c9
Aug 10 15:00:16 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
I20260810 15:00:16.929118 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:16.917390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:33039 (local address 127.28.193.254:38023)
0810 15:00:16.917437 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:16.917439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:16.917450 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:16.918264 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:16.918266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:16.918284 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:16.918321 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:16.919588 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.920256 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.922741 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.923065 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.923115 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:16.925784 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:16.925787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:16.925787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:16.925796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:16.927398 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:16.927921 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:16.927923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:16.927923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:16.927988 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:16.928288 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:16.928289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:16.928289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:16.928437 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:16.928493 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:16.928912 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 15:00:16.929016 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:00:16.929703 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:16.917069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.194:33039)
0810 15:00:16.917270 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:16.917323 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:16.918094 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:16.918889 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:16.918894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:16.919168 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:16.919471 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:16.919479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.920372 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.920373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:16.922629 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:16.922633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:16.923230 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:16.923231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:16.923381 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:16.923943 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:16.923954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:16.925625 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:16.927493 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:16.927502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:16.927505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:16.927825 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:16.928055 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:16.928056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:16.928056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:16.928201 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:16.928533 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:16.928541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:16.928718 (+ 177us) client_negotiation.cc:769] Sending connection context
0810 15:00:16.929272 (+ 554us) client_negotiation.cc:240] Negotiation successful
0810 15:00:16.929453 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260810 15:00:16.930325 17329 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:16.930449 17329 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:16.930666 17329 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:16.931107 16730 ts_manager.cc:194] Registered new tserver with Master: 75f1b599b38246569fd29f0a7072019f (127.28.193.194:44461)
I20260810 15:00:16.931631 16730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:33039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:00:16.952419 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:17.074930 17335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:17.075251 17335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:17.075291 17335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:17.076879 17335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:17.076946 17335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:17.076984 17335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:17.079167 17335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:17.079437 17335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:17.079764 17335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:17.085258 17342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.085281 17343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.085258 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:17.085750 17335 server_base.cc:1034] running on GCE node
I20260810 15:00:17.086017 17335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:17.086246 17335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:17.087424 17335 hybrid_clock.cc:648] HybridClock initialized: now 1786374017087403 us; error 30 us; skew 500 ppm
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:17.090190 17335 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:17.090773 17335 webserver.cc:460] Webserver started at http://127.28.193.195:39245/ using document root <none> and password file <none>
I20260810 15:00:17.091081 17335 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:17.091215 17335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:17.091429 17335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:17.114641 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "6b1b075d25984d6d8099b4d7dfc1ea48"
format_stamp: "Formatted at 2026-08-10 15:00:17 on dist-test-slave-cvwc"
server_key: "1709d0a598fcf6d7ee6f614534748a0d"
server_key_iv: "4bd03655b7ab66602b6a36db76490ce1"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.115319 17335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "6b1b075d25984d6d8099b4d7dfc1ea48"
format_stamp: "Formatted at 2026-08-10 15:00:17 on dist-test-slave-cvwc"
server_key: "1709d0a598fcf6d7ee6f614534748a0d"
server_key_iv: "4bd03655b7ab66602b6a36db76490ce1"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.117244 17335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:17.133023 17358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.133385 17335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:17.133504 17335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "6b1b075d25984d6d8099b4d7dfc1ea48"
format_stamp: "Formatted at 2026-08-10 15:00:17 on dist-test-slave-cvwc"
server_key: "1709d0a598fcf6d7ee6f614534748a0d"
server_key_iv: "4bd03655b7ab66602b6a36db76490ce1"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.133601 17335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:17.159327 17335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:17.161794 17335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:17.161965 17335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:17.162227 17335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:17.162771 17335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:17.162883 17335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.162966 17335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:17.163002 17335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.170622 17335 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:44385
I20260810 15:00:17.170926 17548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:44385 every 8 connection(s)
I20260810 15:00:17.171229 17335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:17.180385 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 17335
I20260810 15:00:17.180488 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:00:17.180675 29447 external_mini_cluster.cc:1428] Setting key 3d23fa8fb2d6dcfdc4454b6f1e5ea027
I20260810 15:00:17.183318 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.172553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:54807 (local address 127.28.193.254:38023)
0810 15:00:17.172604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:17.172606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:17.172763 (+ 157us) server_negotiation.cc:407] Connection header received
0810 15:00:17.173528 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:17.173530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:17.173543 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:17.173622 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:17.174440 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.175221 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.176747 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.177045 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.177093 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.179746 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:17.179747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:17.179748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:17.179757 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:17.181549 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.182118 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.182120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.182120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.182189 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.182526 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.182527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.182527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.182670 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:17.182741 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:17.183125 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 15:00:17.183221 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:00:17.183883 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.172236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.195:54807)
0810 15:00:17.172599 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:17.172649 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:17.173387 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:17.173702 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:17.173707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:17.173999 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:17.174288 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.174296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.175339 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.175340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.176625 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.176630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.177207 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.177208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.177350 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.177969 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:17.177979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:17.179653 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:17.181650 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.181657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.181659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.182012 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.182277 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.182278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.182278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.182434 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.182779 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:17.182786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:17.182947 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 15:00:17.183466 (+ 519us) client_negotiation.cc:240] Negotiation successful
0810 15:00:17.183635 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":84,"threads_started":1}
I20260810 15:00:17.184347 17551 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:17.184451 17551 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.184649 17551 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:17.185048 16730 ts_manager.cc:194] Registered new tserver with Master: 6b1b075d25984d6d8099b4d7dfc1ea48 (127.28.193.195:44385)
I20260810 15:00:17.185526 16730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:54807
I20260810 15:00:17.192384 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:00:17.193249 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 16880
I20260810 15:00:17.217576 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 17108
I20260810 15:00:17.243706 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 17335
I20260810 15:00:17.282431 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 16675
I20260810 15:00:17.307320 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:38023
--webserver_interface=127.28.193.254
--webserver_port=37945
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:38023
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:17.390791 17557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:17.391000 17557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:17.391022 17557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:17.392154 17557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:17.392203 17557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:17.392226 17557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:17.392241 17557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:17.392269 17557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:17.393891 17557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:38023
--ranger_config_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=37945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:17.394121 17557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:17.394335 17557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:17.399063 17570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.399075 17567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.399129 17568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:17.399362 17557 server_base.cc:1034] running on GCE node
I20260810 15:00:17.399642 17557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:17.399919 17557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:17.401073 17557 hybrid_clock.cc:648] HybridClock initialized: now 1786374017401054 us; error 35 us; skew 500 ppm
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:17.403466 17557 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:17.403849 17557 webserver.cc:460] Webserver started at http://127.28.193.254:37945/ using document root <none> and password file <none>
I20260810 15:00:17.404016 17557 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:17.404062 17557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:17.473974 17557 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:00:17.474820 17585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.474990 17557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:17.475080 17557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "79776cb8c32344c28495b9d469922de6"
format_stamp: "Formatted at 2026-08-10 15:00:15 on dist-test-slave-cvwc"
server_key: "d2f8b881a35dc0df23ba93e4023a5fb3"
server_key_iv: "14db4b80b4fd65bba1fa6b61ec803eda"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.475461 17557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:17.486132 17557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:17.487895 17557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:17.488018 17557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:17.492558 17557 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:38023
I20260810 15:00:17.492614 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:38023 every 8 connection(s)
I20260810 15:00:17.492983 17557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:17.493422 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 17557
I20260810 15:00:17.493666 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:34141
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:38023
--webserver_port=34815
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:17.494346 17679 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:00:17.494977 17679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Bootstrap starting.
I20260810 15:00:17.503942 17679 log.cc:824] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:17.505585 17679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:00:17.505865 17679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Bootstrap complete.
I20260810 15:00:17.508153 17679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:17.508401 17679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79776cb8c32344c28495b9d469922de6, State: Initialized, Role: FOLLOWER
I20260810 15:00:17.508559 17679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:17.508635 17679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:17.508682 17679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:17.508728 17679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:00:17.509757 17679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:17.509858 17679 leader_election.cc:302] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79776cb8c32344c28495b9d469922de6; no voters:
I20260810 15:00:17.510028 17679 leader_election.cc:288] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:00:17.510071 17685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:00:17.510188 17685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [term 2 LEADER]: Becoming Leader. State: Replica: 79776cb8c32344c28495b9d469922de6, State: Running, Role: LEADER
I20260810 15:00:17.510301 17679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:17.510289 17685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } }
I20260810 15:00:17.510603 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "79776cb8c32344c28495b9d469922de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } } }
I20260810 15:00:17.510736 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:17.510609 17685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79776cb8c32344c28495b9d469922de6. Latest consensus state: current_term: 2 leader_uuid: "79776cb8c32344c28495b9d469922de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79776cb8c32344c28495b9d469922de6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 38023 } } }
I20260810 15:00:17.510876 17685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:17.511080 17704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:17.511600 17704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:17.511729 17704 catalog_manager.cc:1259] Loaded cluster ID: ea8e7dd7685a414ca501546d20cca252
I20260810 15:00:17.511780 17704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:00:17.511951 17679 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:00:17.516304 17704 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:17.516986 17704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Loaded TSK: 0
I20260810 15:00:17.517274 17704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:00:17.596058 17683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:17.596252 17683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:17.596284 17683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:17.597555 17683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:17.597675 17683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:17.597759 17683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:17.599423 17683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=34815
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:17.599660 17683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:17.599917 17683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:17.605293 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:17.605458 17683 server_base.cc:1034] running on GCE node
W20260810 15:00:17.605929 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.606132 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:17.606361 17683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:17.606590 17683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:17.607753 17683 hybrid_clock.cc:648] HybridClock initialized: now 1786374017607742 us; error 28 us; skew 500 ppm
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:17.610986 17683 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:17.611493 17683 webserver.cc:460] Webserver started at http://127.28.193.193:34815/ using document root <none> and password file <none>
I20260810 15:00:17.611730 17683 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:17.611793 17683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:17.637663 17683 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 15:00:17.641474 17771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.641770 17683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260810 15:00:17.641873 17683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "486762e5a42b446fbb3959a310cd9c70"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "b10d3aba5e5fc90e57c0c99359714b8a"
server_key_iv: "593b079c111ee6819bc68406f80ca57b"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.642372 17683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:17.670212 17683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:17.672399 17683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:17.672559 17683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:17.672835 17683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:17.673273 17683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:17.673378 17683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.673491 17683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:17.673569 17683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.684104 17683 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:34141
I20260810 15:00:17.684619 17683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:00:17.685891 17980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:34141 every 8 connection(s)
I20260810 15:00:17.690053 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 17683
I20260810 15:00:17.690393 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:44461
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:38023
--webserver_port=35863
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:17.699851 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.685475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.193:32841)
0810 15:00:17.686103 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:17.686146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:17.686788 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:17.687257 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:17.687261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:17.687453 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:17.687643 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.687648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.689121 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.689122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.690050 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.690054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.692953 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.692955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.693115 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.693704 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:17.693741 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:17.695437 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:17.697871 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.697880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.697883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.698182 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.698497 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.698498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.698498 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.698638 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.699137 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:17.699143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:17.699307 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 15:00:17.699422 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:00:17.699592 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":86,"threads_started":1}
I20260810 15:00:17.699875 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.685561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:32841 (local address 127.28.193.254:38023)
0810 15:00:17.686842 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:17.686853 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:17.686897 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:00:17.687040 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:17.687045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:17.687088 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:17.687205 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:17.687764 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.689011 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.692150 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.692810 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.692930 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.695575 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:17.695581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:17.695581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:17.695596 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:17.697769 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.698284 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.698289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.698289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.698412 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.698714 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.698715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.698716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.699055 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:17.699145 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:17.699518 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 15:00:17.699652 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1217,"thread_start_us":95,"threads_started":1}
I20260810 15:00:17.701217 17982 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:17.701331 17982 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.701586 17982 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:17.702960 17982 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:38023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:00:17.703064 17982 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.703163 17982 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:17.703840 17982 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.703935 17982 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:17.713697 17982 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:00:17.802485 17987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:17.802706 17987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:17.802736 17987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:17.803900 17987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:17.803939 17987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:17.803974 17987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:17.805435 17987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:44461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=35863
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:17.805670 17987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:17.805924 17987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:17.811092 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.811074 18002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:17.811136 18000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:17.811498 17987 server_base.cc:1034] running on GCE node
I20260810 15:00:17.811822 17987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:17.812014 17987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:17.813150 17987 hybrid_clock.cc:648] HybridClock initialized: now 1786374017813127 us; error 39 us; skew 500 ppm
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:17.815634 17987 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:17.816061 17987 webserver.cc:460] Webserver started at http://127.28.193.194:35863/ using document root <none> and password file <none>
I20260810 15:00:17.816291 17987 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:17.816355 17987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:17.835196 17987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:17.836306 18016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.836519 17987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:17.836619 17987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "75f1b599b38246569fd29f0a7072019f"
format_stamp: "Formatted at 2026-08-10 15:00:16 on dist-test-slave-cvwc"
server_key: "85fc71252a8e36b02eedd771f64f3ee3"
server_key_iv: "8609234f0a009dc9660c66ad0df28ac9"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:17.837113 17987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:17.868728 17987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:17.871147 17987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:17.871325 17987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:17.871613 17987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:17.872030 17987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:17.872082 17987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.872123 17987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:17.872148 17987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:17.879732 17987 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44461
I20260810 15:00:17.879908 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44461 every 8 connection(s)
I20260810 15:00:17.880229 17987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:00:17 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374017, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:17.889485 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.881024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:38761 (local address 127.28.193.254:38023)
0810 15:00:17.881076 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:17.881078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:17.881347 (+ 269us) server_negotiation.cc:407] Connection header received
0810 15:00:17.882003 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:17.882005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:17.882020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:17.882082 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:17.882661 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.883163 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.884163 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.884388 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.884453 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.886734 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:17.886736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:17.886736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:17.886746 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:17.888096 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.888504 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.888506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.888506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.888577 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:17.888839 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:17.888840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:17.888840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:17.888974 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:17.889034 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:17.889298 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 15:00:17.889390 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:00:17.889854 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:17.881009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.194:38761)
0810 15:00:17.881188 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:17.881237 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:17.881885 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:17.882159 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:17.882163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:17.882352 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:17.882559 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.882564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.883259 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.883260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.884071 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:17.884074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:17.884498 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:17.884500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:17.884618 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:17.885144 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:17.885153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:17.886632 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:17.888187 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.888195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.888197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.888415 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.888648 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:17.888649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:17.888649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:17.888773 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:17.889033 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:17.889041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:17.889169 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 15:00:17.889258 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:00:17.889400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260810 15:00:17.889858 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 17987
I20260810 15:00:17.890166 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.193.212:51427/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:44385
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:38023
--webserver_port=39245
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:38759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:17.890342 18216 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:17.890441 18216 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.890618 18216 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:17.891249 18216 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:38023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:00:17.891314 18216 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.891379 18216 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:17.891661 18216 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:17.891741 18216 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:17.892038 18216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:00:18.002630 18220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:18.002907 18220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:18.002944 18220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:18.004804 18220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:18.004863 18220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:18.004899 18220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:18.007040 18220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:38759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.193.212:51427/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:44385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=39245
--tserver_master_addrs=127.28.193.254:38023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:18.007301 18220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:18.007568 18220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:18.014056 18233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:18.014402 18220 server_base.cc:1034] running on GCE node
W20260810 15:00:18.014585 18231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:18.014739 18229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:18.014989 18220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:18.015265 18220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:18.016976 18220 hybrid_clock.cc:648] HybridClock initialized: now 1786374018016946 us; error 42 us; skew 500 ppm
Aug 10 15:00:18 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:18.020212 18220 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:18.020674 18220 webserver.cc:460] Webserver started at http://127.28.193.195:39245/ using document root <none> and password file <none>
I20260810 15:00:18.020957 18220 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:18.021030 18220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:00:18 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:18.036584 18220 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:18.037690 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:18.037899 18220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:18.037987 18220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "6b1b075d25984d6d8099b4d7dfc1ea48"
format_stamp: "Formatted at 2026-08-10 15:00:17 on dist-test-slave-cvwc"
server_key: "1709d0a598fcf6d7ee6f614534748a0d"
server_key_iv: "4bd03655b7ab66602b6a36db76490ce1"
server_key_version: "kuduclusterkey@0"
I20260810 15:00:18.038353 18220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:18.084363 18220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:18.086388 18220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:18.086555 18220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:18.086830 18220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:18.087173 18220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:18.087206 18220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:18.087229 18220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:18.087245 18220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:18.094327 18220 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:44385
I20260810 15:00:18.094717 18220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:00:18.095141 18461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:44385 every 8 connection(s)
I20260810 15:00:18.097556 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 18220
Aug 10 15:00:18 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:18.104681 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:18.095488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:46045 (local address 127.28.193.254:38023)
0810 15:00:18.095541 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:18.095543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:18.095749 (+ 206us) server_negotiation.cc:407] Connection header received
0810 15:00:18.096412 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:18.096414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:18.096427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:18.096459 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:18.097119 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.097804 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.098896 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.099193 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.099240 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:18.101583 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:18.101585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:18.101585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:18.101593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:18.103266 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:18.103740 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:18.103742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:18.103742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:18.103807 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:18.104077 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:18.104078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:18.104078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:18.104191 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:18.104249 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:18.104506 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 15:00:18.104592 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:00:18.104843 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:18.095456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:38023 (local address 127.28.193.195:46045)
0810 15:00:18.095607 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:18.095650 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:18.096279 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:18.096641 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:18.096645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:18.096830 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:18.097026 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:18.097030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.097899 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.097900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:18.098786 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:18.098789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.099355 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.099357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:18.099495 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:18.100049 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:18.100059 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:18.101489 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:18.103359 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:18.103366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:18.103368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:18.103649 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:18.103875 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:18.103876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:18.103877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:18.103990 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:18.104255 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:18.104262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:18.104386 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 15:00:18.104476 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:00:18.104613 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260810 15:00:18.105456 18462 heartbeater.cc:347] Connected to a master server at 127.28.193.254:38023
I20260810 15:00:18.105557 18462 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:18.105779 18462 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:18.106577 18462 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:38023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:00:18.106706 18462 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:18.106827 18462 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:18.107230 18462 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:18.107311 18462 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
W20260810 15:00:18.107823 18462 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:00:18.109737 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:18.097984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.28.193.254:38023)
0810 15:00:18.100651 (+ 2667us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:18.100652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:18.100661 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:00:18.100681 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:18.100682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:18.100693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:18.100772 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:18.101755 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.102460 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.105342 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:18.105649 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:18.105763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:18.107066 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:18.107137 (+ 71us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:18.107138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:18.107144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:18.108677 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:18.108985 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:18.108986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:18.108987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:18.109048 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:18.109212 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:18.109213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:18.109213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:18.109330 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:18.109497 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:18.109538 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 15:00:18.109619 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":83,"threads_started":1}
Aug 10 15:00:18 dist-test-slave-cvwc krb5kdc[15925](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:18.713995 17982 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:18.714169 17982 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:18.714614 17618 ts_manager.cc:194] Registered new tserver with Master: 486762e5a42b446fbb3959a310cd9c70 (127.28.193.193:34141)
I20260810 15:00:18.715634 17618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:32841
I20260810 15:00:18.892235 18216 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:18.892400 18216 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:18.892699 17618 ts_manager.cc:194] Registered new tserver with Master: 75f1b599b38246569fd29f0a7072019f (127.28.193.194:44461)
I20260810 15:00:18.893167 17618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:38761
Aug 10 15:00:18 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:19.108011 18462 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:19.108193 18462 heartbeater.cc:511] Master 127.28.193.254:38023 requested a full tablet report, sending...
I20260810 15:00:19.108603 17618 ts_manager.cc:194] Registered new tserver with Master: 6b1b075d25984d6d8099b4d7dfc1ea48 (127.28.193.195:44385)
I20260810 15:00:19.109151 17618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:46045
I20260810 15:00:19.114075 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:00:19.121423 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.114952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.28.193.254:38023)
0810 15:00:19.115052 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.115054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.115106 (+ 52us) server_negotiation.cc:407] Connection header received
0810 15:00:19.115171 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.115172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.115185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.115242 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:19.115781 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.116287 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.117088 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.117618 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.118113 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.119067 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:19.119069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:19.119069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:19.119076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:19.120435 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:19.120737 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:19.120739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:19.120739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:19.120794 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:19.120957 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:19.120958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:19.120959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:19.121061 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:19.121149 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.121245 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.121325 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260810 15:00:19.123377 17618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:00:19.125072 17618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:00:19.140106 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.127804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44461 (local address 127.0.0.1:40238)
0810 15:00:19.133876 (+ 6072us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.133886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:19.133924 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:19.136096 (+ 2172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:19.136097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:19.136099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:19.136160 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.136162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.137539 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.137540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.138701 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.138704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.139784 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.139785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.139891 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.139909 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 15:00:19.139961 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:00:19.140011 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6054,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260810 15:00:19.140568 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.128239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.28.193.194:44461)
0810 15:00:19.135791 (+ 7552us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.135795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.135806 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:19.135841 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.135843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.135955 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.136240 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:19.136347 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.137432 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.138819 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.139673 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.140340 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.140416 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.140444 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.140488 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7504,"thread_start_us":56,"threads_started":1}
I20260810 15:00:19.140781 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.128052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:34141 (local address 127.0.0.1:59124)
0810 15:00:19.133994 (+ 5942us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.134003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:19.134041 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:19.134330 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:19.134331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:19.134333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:19.134383 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.134384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.135587 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.135588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.138757 (+ 3169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.138759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.140552 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.140553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.140625 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.140629 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:19.140679 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:00:19.140717 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5929,"mutex_wait_us":911,"thread_start_us":63,"threads_started":1}
I20260810 15:00:19.141112 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.128166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.28.193.193:34141)
0810 15:00:19.128309 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.128315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.134100 (+ 5785us) server_negotiation.cc:407] Connection header received
0810 15:00:19.134143 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.134146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.134162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.134231 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:19.134460 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.135506 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.138851 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.140100 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.140148 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.140490 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.140981 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.141036 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260810 15:00:19.141562 18108 tablet_service.cc:1467] Processing CreateTablet for tablet e179f251d9044ffb874509175d11875f (DEFAULT_TABLE table=test-table [id=fcb40cfafc55415bbb345403a2d2ae12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:19.142017 18108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179f251d9044ffb874509175d11875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:19.142473 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.128031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.28.193.195:44385)
0810 15:00:19.133476 (+ 5445us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.133482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.133743 (+ 261us) server_negotiation.cc:407] Connection header received
0810 15:00:19.133834 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.133837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.133856 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.133930 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:19.134220 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.135418 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.137679 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.141949 (+ 4270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.142020 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.142121 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.142353 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.142394 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5398,"thread_start_us":74,"threads_started":1}
I20260810 15:00:19.142724 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.127804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:44385 (local address 127.0.0.1:44958)
0810 15:00:19.133624 (+ 5820us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.133663 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:19.133750 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:19.134052 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:19.134056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:19.134058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:19.134140 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.134142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.135519 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.135520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.137498 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.137502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.142029 (+ 4527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.142029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.142176 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.142180 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:19.142234 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:00:19.142275 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5787,"mutex_wait_us":4,"thread_start_us":94,"threads_started":1}
I20260810 15:00:19.142632 17842 tablet_service.cc:1467] Processing CreateTablet for tablet e179f251d9044ffb874509175d11875f (DEFAULT_TABLE table=test-table [id=fcb40cfafc55415bbb345403a2d2ae12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:19.143066 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179f251d9044ffb874509175d11875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:19.143810 18328 tablet_service.cc:1467] Processing CreateTablet for tablet e179f251d9044ffb874509175d11875f (DEFAULT_TABLE table=test-table [id=fcb40cfafc55415bbb345403a2d2ae12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:19.144157 18328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e179f251d9044ffb874509175d11875f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:19.145874 18518 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: Bootstrap starting.
I20260810 15:00:19.146571 18518 tablet_bootstrap.cc:654] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:19.146934 18518 log.cc:824] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:19.147863 18518 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: No bootstrap required, opened a new log
I20260810 15:00:19.147953 18518 ts_tablet_manager.cc:1397] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:19.150652 18518 raft_consensus.cc:348] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.150842 18518 raft_consensus.cc:374] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:19.150879 18518 raft_consensus.cc:729] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f1b599b38246569fd29f0a7072019f, State: Initialized, Role: FOLLOWER
I20260810 15:00:19.151058 18518 consensus_queue.cc:260] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.151391 18518 ts_tablet_manager.cc:1428] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:00:19.151522 18216 heartbeater.cc:503] Master 127.28.193.254:38023 was elected leader, sending a full tablet report...
I20260810 15:00:19.153898 18520 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Bootstrap starting.
I20260810 15:00:19.154630 18520 tablet_bootstrap.cc:654] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:19.154961 18520 log.cc:824] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:19.155381 18524 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Bootstrap starting.
I20260810 15:00:19.156064 18524 tablet_bootstrap.cc:654] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:19.156332 18524 log.cc:824] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:19.157222 18524 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: No bootstrap required, opened a new log
I20260810 15:00:19.157299 18524 ts_tablet_manager.cc:1397] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:19.159534 18524 raft_consensus.cc:348] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.159722 18524 raft_consensus.cc:374] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:19.159754 18524 raft_consensus.cc:729] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1b075d25984d6d8099b4d7dfc1ea48, State: Initialized, Role: FOLLOWER
I20260810 15:00:19.159916 18524 consensus_queue.cc:260] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.160224 18524 ts_tablet_manager.cc:1428] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:00:19.160351 18462 heartbeater.cc:503] Master 127.28.193.254:38023 was elected leader, sending a full tablet report...
I20260810 15:00:19.162581 18520 tablet_bootstrap.cc:492] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: No bootstrap required, opened a new log
I20260810 15:00:19.162667 18520 ts_tablet_manager.cc:1397] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 15:00:19.164865 18520 raft_consensus.cc:348] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.165040 18520 raft_consensus.cc:374] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:19.165071 18520 raft_consensus.cc:729] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486762e5a42b446fbb3959a310cd9c70, State: Initialized, Role: FOLLOWER
I20260810 15:00:19.165192 18520 consensus_queue.cc:260] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.165474 18520 ts_tablet_manager.cc:1428] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:00:19.165525 17982 heartbeater.cc:503] Master 127.28.193.254:38023 was elected leader, sending a full tablet report...
W20260810 15:00:19.186801 17984 tablet.cc:2369] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:19.210289 18533 raft_consensus.cc:482] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:19.210459 18533 raft_consensus.cc:504] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.210892 18533 leader_election.cc:288] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f1b599b38246569fd29f0a7072019f (127.28.193.194:44461), 486762e5a42b446fbb3959a310cd9c70 (127.28.193.193:34141)
I20260810 15:00:19.215404 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.211490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:49607 (local address 127.28.193.193:34141)
0810 15:00:19.211566 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.211567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.211694 (+ 127us) server_negotiation.cc:407] Connection header received
0810 15:00:19.211800 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.211801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.211810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.211841 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:19.212080 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.212991 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.214452 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.215010 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.215061 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.215071 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.215290 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.215330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:00:19.215454 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.211205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:43539 (local address 127.28.193.194:44461)
0810 15:00:19.211266 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.211268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.211659 (+ 391us) server_negotiation.cc:407] Connection header received
0810 15:00:19.211793 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.211795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.211807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.211842 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:19.212100 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.212877 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.214447 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.215032 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.215083 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.215091 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.215337 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.215380 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:00:19.215708 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.211108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44461 (local address 127.28.193.195:43539)
0810 15:00:19.211583 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.211595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:19.211691 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:19.211918 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:19.211919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:19.211921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:19.212020 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.212022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.212986 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.212987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.214344 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.214347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.215172 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.215173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.215253 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.215257 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:19.215568 (+ 311us) client_negotiation.cc:240] Negotiation successful
0810 15:00:19.215614 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":44,"thread_start_us":136,"threads_started":1}
I20260810 15:00:19.216007 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.211389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:34141 (local address 127.28.193.195:49607)
0810 15:00:19.211622 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.211633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:19.211720 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:19.211919 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:19.211920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:19.211924 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:19.212010 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.212012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.213095 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.213095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.214343 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:19.214346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.215149 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.215150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:19.215217 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.215221 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:19.215880 (+ 659us) client_negotiation.cc:240] Negotiation successful
0810 15:00:19.215923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":17,"thread_start_us":105,"threads_started":1}
I20260810 15:00:19.216043 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179f251d9044ffb874509175d11875f" candidate_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f1b599b38246569fd29f0a7072019f" is_pre_election: true
I20260810 15:00:19.216225 18116 raft_consensus.cc:2454] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1b075d25984d6d8099b4d7dfc1ea48 in term 0.
I20260810 15:00:19.216322 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179f251d9044ffb874509175d11875f" candidate_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486762e5a42b446fbb3959a310cd9c70" is_pre_election: true
I20260810 15:00:19.216482 17905 raft_consensus.cc:2454] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b1b075d25984d6d8099b4d7dfc1ea48 in term 0.
I20260810 15:00:19.216487 18255 leader_election.cc:302] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1b075d25984d6d8099b4d7dfc1ea48, 75f1b599b38246569fd29f0a7072019f; no voters:
I20260810 15:00:19.216678 18533 raft_consensus.cc:2781] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:00:19.216763 18533 raft_consensus.cc:482] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:00:19.216797 18533 raft_consensus.cc:3037] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:19.217774 18533 raft_consensus.cc:504] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.217922 18533 leader_election.cc:288] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [CANDIDATE]: Term 1 election: Requested vote from peers 75f1b599b38246569fd29f0a7072019f (127.28.193.194:44461), 486762e5a42b446fbb3959a310cd9c70 (127.28.193.193:34141)
I20260810 15:00:19.218123 17905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179f251d9044ffb874509175d11875f" candidate_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "486762e5a42b446fbb3959a310cd9c70"
I20260810 15:00:19.218137 18116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e179f251d9044ffb874509175d11875f" candidate_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f1b599b38246569fd29f0a7072019f"
I20260810 15:00:19.218202 17905 raft_consensus.cc:3037] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:19.218217 18116 raft_consensus.cc:3037] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:19.219146 17905 raft_consensus.cc:2454] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1b075d25984d6d8099b4d7dfc1ea48 in term 1.
I20260810 15:00:19.219215 18116 raft_consensus.cc:2454] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b1b075d25984d6d8099b4d7dfc1ea48 in term 1.
I20260810 15:00:19.219388 18255 leader_election.cc:302] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b1b075d25984d6d8099b4d7dfc1ea48, 75f1b599b38246569fd29f0a7072019f; no voters:
I20260810 15:00:19.219461 18533 raft_consensus.cc:2781] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:19.219609 18533 raft_consensus.cc:686] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 1 LEADER]: Becoming Leader. State: Replica: 6b1b075d25984d6d8099b4d7dfc1ea48, State: Running, Role: LEADER
I20260810 15:00:19.219712 18533 consensus_queue.cc:237] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } }
I20260810 15:00:19.220476 17618 catalog_manager.cc:5495] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b1b075d25984d6d8099b4d7dfc1ea48 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b1b075d25984d6d8099b4d7dfc1ea48" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44385 } health_report { overall_health: HEALTHY } } }
I20260810 15:00:19.247906 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.244665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.28.193.195:44385)
0810 15:00:19.244759 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:19.244761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:19.244768 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:00:19.244806 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:19.244807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:19.244826 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:19.244900 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:00:19.245089 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.245823 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.246818 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:19.247093 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:19.247171 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:19.247376 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:00:19.247509 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:00:19.247583 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:19.247799 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 15:00:19.247820 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 15:00:19.250079 17905 raft_consensus.cc:1261] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 1 FOLLOWER]: Refusing update from remote peer 6b1b075d25984d6d8099b4d7dfc1ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:19.250475 18533 consensus_queue.cc:1035] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486762e5a42b446fbb3959a310cd9c70" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:19.252395 18116 raft_consensus.cc:1261] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 1 FOLLOWER]: Refusing update from remote peer 6b1b075d25984d6d8099b4d7dfc1ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:19.252717 18533 consensus_queue.cc:1035] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f1b599b38246569fd29f0a7072019f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:19.255735 18557 mvcc.cc:205] Tried to move back new op lower bound from 7316987982845202432 to 7316987982724665344. Current Snapshot: MvccSnapshot[applied={T|T < 7316987982845202432}]
I20260810 15:00:19.256983 18554 mvcc.cc:205] Tried to move back new op lower bound from 7316987982845202432 to 7316987982724665344. Current Snapshot: MvccSnapshot[applied={T|T < 7316987982845202432}]
I20260810 15:00:19.259560 17618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:19.259673 17618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:19.260465 17618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79776cb8c32344c28495b9d469922de6: Sending DeleteTablet for 3 replicas of tablet e179f251d9044ffb874509175d11875f
I20260810 15:00:19.260833 18328 tablet_service.cc:1514] Processing DeleteTablet for tablet e179f251d9044ffb874509175d11875f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:19 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:44958
I20260810 15:00:19.261065 17842 tablet_service.cc:1514] Processing DeleteTablet for tablet e179f251d9044ffb874509175d11875f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:19 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:59124
I20260810 15:00:19.261288 18565 tablet_replica.cc:330] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: stopping tablet replica
I20260810 15:00:19.261426 18565 raft_consensus.cc:2229] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:19.261509 18108 tablet_service.cc:1514] Processing DeleteTablet for tablet e179f251d9044ffb874509175d11875f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:19 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:40238
I20260810 15:00:19.261665 18565 raft_consensus.cc:2258] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:19.261981 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 17683
I20260810 15:00:19.262215 18565 ts_tablet_manager.cc:1905] T e179f251d9044ffb874509175d11875f P 486762e5a42b446fbb3959a310cd9c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:19.274427 18553 mvcc.cc:205] Tried to move back new op lower bound from 7316987982845202432 to 7316987982724665344. Current Snapshot: MvccSnapshot[applied={T|T < 7316987982845202432 or (T in {7316987982845202432})}]
I20260810 15:00:19.276950 18566 tablet_replica.cc:330] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f: stopping tablet replica
I20260810 15:00:19.277100 18566 raft_consensus.cc:2229] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:19.277232 18566 raft_consensus.cc:2258] T e179f251d9044ffb874509175d11875f P 75f1b599b38246569fd29f0a7072019f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:19.277411 18564 tablet_replica.cc:330] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: stopping tablet replica
I20260810 15:00:19.277518 18564 raft_consensus.cc:2229] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:00:19.277645 18564 raft_consensus.cc:2258] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 15:00:19.279021 17591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:34141 (error 108)
I20260810 15:00:19.279750 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 17987
Aug 10 15:00:19 dist-test-slave-cvwc krb5kdc[15925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for HTTP/127.28.193.212@KRBTEST.COM
I20260810 15:00:19.294585 18564 ts_tablet_manager.cc:1905] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:19.295727 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.295564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:34141 (local address 127.0.0.1:59126)
0810 15:00:19.295626 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.295686 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:34141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:00:19.295886 17591 catalog_manager.cc:4584] TS 486762e5a42b446fbb3959a310cd9c70 (127.28.193.193:34141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e179f251d9044ffb874509175d11875f: Network error: Client connection negotiation failed: client connection to 127.28.193.193:34141: connect: Connection refused (error 111)
I20260810 15:00:19.296583 18564 ts_tablet_manager.cc:1918] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:19.296674 18564 log.cc:1196] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/e179f251d9044ffb874509175d11875f
I20260810 15:00:19.296947 18564 ts_tablet_manager.cc:1939] T e179f251d9044ffb874509175d11875f P 6b1b075d25984d6d8099b4d7dfc1ea48: Deleting consensus metadata
I20260810 15:00:19.297415 17594 catalog_manager.cc:4841] TS 6b1b075d25984d6d8099b4d7dfc1ea48 (127.28.193.195:44385): tablet e179f251d9044ffb874509175d11875f (table test-table [id=fcb40cfafc55415bbb345403a2d2ae12]) successfully deleted
I20260810 15:00:19.301219 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:19.301116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44461 (local address 127.0.0.1:40246)
0810 15:00:19.301163 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:19.301186 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.194:44461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:00:19.301273 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 18220
I20260810 15:00:19.316773 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 17557
I20260810 15:00:19.329546 29447 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 15:00:19.674302 29447 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 15:00:19.674399 29447 mini_ranger.cc:66] Stopping Ranger...
I20260810 15:00:20.018461 29447 mini_ranger.cc:68] Stopped Ranger
2026-08-10 15:00:20.018 UTC [15977] LOG: received smart shutdown request
2026-08-10 15:00:20.023 UTC [15977] LOG: background worker "logical replication launcher" (PID 15989) exited with exit code 1
2026-08-10 15:00:20.023 UTC [15982] LOG: shutting down
2026-08-10 15:00:20.158 UTC [15977] LOG: database system is shut down
2026-08-10T15:00:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45967 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:00:20 dist-test-slave-cvwc krb5kdc[18588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:00:20 dist-test-slave-cvwc krb5kdc[18588](info): set up 2 sockets
Aug 10 15:00:20 dist-test-slave-cvwc krb5kdc[18588](info): commencing operation
krb5kdc: starting...
W20260810 15:00:21.410404 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:00:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:00:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:00:21.528661 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:40367
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:21.610153 18607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:21.610368 18607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:21.610388 18607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:21.611547 18607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:21.611594 18607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:21.611608 18607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:21.611619 18607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:21.611629 18607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:21.613170 18607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:21.613409 18607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:21.613641 18607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:21.618615 18614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:21.618615 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:21.618723 18607 server_base.cc:1034] running on GCE node
W20260810 15:00:21.618645 18616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:21.619104 18607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:21.619403 18607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:21.620546 18607 hybrid_clock.cc:648] HybridClock initialized: now 1786374021620533 us; error 25 us; skew 500 ppm
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:21.622982 18607 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:21.623426 18607 webserver.cc:460] Webserver started at http://127.28.193.254:44641/ using document root <none> and password file <none>
I20260810 15:00:21.623644 18607 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:21.623693 18607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:21.623822 18607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:21.624878 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "fcccbbc8b2c54a148d6e8776a6963b75"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "776b2957ff0199d96b69c83aa91ec5fb"
server_key_iv: "a68554d71b5ad226ae423eb89ffd7d89"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.625211 18607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "fcccbbc8b2c54a148d6e8776a6963b75"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "776b2957ff0199d96b69c83aa91ec5fb"
server_key_iv: "a68554d71b5ad226ae423eb89ffd7d89"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.626982 18607 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260810 15:00:21.627712 18631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:21.627910 18607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:21.627996 18607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "fcccbbc8b2c54a148d6e8776a6963b75"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "776b2957ff0199d96b69c83aa91ec5fb"
server_key_iv: "a68554d71b5ad226ae423eb89ffd7d89"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.628062 18607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:21.639858 18607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:21.641522 18607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:21.641651 18607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:21.645551 18607 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:40367
I20260810 15:00:21.645588 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:40367 every 8 connection(s)
I20260810 15:00:21.645949 18607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:21.646826 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:21.648399 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:00:21.653334 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 18607
I20260810 15:00:21.653457 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:00:21.653621 29447 external_mini_cluster.cc:1428] Setting key 5d41037dd52bb3f34143e2108334efd1
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:21.657346 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.649038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40367 (local address 127.0.0.1:33664)
0810 15:00:21.649214 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.649259 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.649802 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.650069 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.650072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.650347 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.650498 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.650503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.651470 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.651471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.652126 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.652128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.652588 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.652589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.652653 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.653086 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.653094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.654448 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.656027 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.656033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.656035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.656212 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.656415 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.656415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.656416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.656492 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.656801 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.656807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.656872 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.656957 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.657177 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":88,"thread_start_us":84,"threads_started":1}
I20260810 15:00:21.657346 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.649097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.28.193.254:40367)
0810 15:00:21.649259 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.649273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.649365 (+ 92us) server_negotiation.cc:407] Connection header received
0810 15:00:21.649952 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.649954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.649975 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.650060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.650592 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.651385 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.652200 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.652516 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.652608 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.654524 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.654531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.654531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.654537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.655952 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.656279 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.656282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.656283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.656357 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.656544 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.656545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.656545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.656740 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.656837 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.656988 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.657104 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260810 15:00:21.658313 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:00:21.658701 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.658562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47128)
0810 15:00:21.658610 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.658677 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 15:00:21.658870 18639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
I20260810 15:00:21.659282 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.659201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47132)
0810 15:00:21.659239 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.659263 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:00:21.659394 18723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:38081: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
I20260810 15:00:21.680238 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:38081
--webserver_interface=127.28.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:21.708652 18723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } attempt: 1
I20260810 15:00:21.709206 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.709059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47134)
0810 15:00:21.709121 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.709171 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:00:21.709587 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.709494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47142)
0810 15:00:21.709538 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.709567 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:00:21.709702 18723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:38081: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
W20260810 15:00:21.761590 18734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:21.761824 18734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:21.761848 18734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:21.762974 18734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:21.763013 18734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:21.763026 18734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:21.763036 18734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:21.763047 18734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:21.764564 18734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:21.764827 18734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:21.765072 18734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:21.769973 18750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:21.769979 18745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:21.769986 18744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:21.770196 18734 server_base.cc:1034] running on GCE node
I20260810 15:00:21.770515 18734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:21.770793 18734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:21.771965 18734 hybrid_clock.cc:648] HybridClock initialized: now 1786374021771947 us; error 35 us; skew 500 ppm
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:21.774536 18734 init.cc:377] Logged in from keytab as kudu/127.28.193.253@KRBTEST.COM (short username kudu)
I20260810 15:00:21.774962 18734 webserver.cc:460] Webserver started at http://127.28.193.253:38809/ using document root <none> and password file <none>
I20260810 15:00:21.775175 18734 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:21.775228 18734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:21.775326 18734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:21.776342 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data/instance:
uuid: "74d282bb4ab346458b0f75bf8a94adf7"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "f1ea7ca34fcd80d282195fbeee29b97f"
server_key_iv: "f76c29c2729814db21d6f08530878f7b"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.776669 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal/instance:
uuid: "74d282bb4ab346458b0f75bf8a94adf7"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "f1ea7ca34fcd80d282195fbeee29b97f"
server_key_iv: "f76c29c2729814db21d6f08530878f7b"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.777890 18723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } attempt: 2
I20260810 15:00:21.778075 18734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:00:21.778275 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.778166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47144)
0810 15:00:21.778209 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.778240 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 15:00:21.778609 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.778519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47150)
0810 15:00:21.778560 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.778585 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:00:21.778762 18723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:38081: Network error: Client connection negotiation failed: client connection to 127.28.193.253:38081: connect: Connection refused (error 111)
I20260810 15:00:21.778946 18759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:21.779089 18734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:21.779171 18734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "74d282bb4ab346458b0f75bf8a94adf7"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "f1ea7ca34fcd80d282195fbeee29b97f"
server_key_iv: "f76c29c2729814db21d6f08530878f7b"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.779224 18734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:21.790205 18734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:21.791813 18734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:21.791917 18734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:21.795524 18734 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.253:38081
I20260810 15:00:21.795568 18855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.253:38081 every 8 connection(s)
I20260810 15:00:21.795876 18734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
I20260810 15:00:21.796728 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:21.798321 18857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:21.805274 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 18734
I20260810 15:00:21.805374 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-1/wal/instance
I20260810 15:00:21.805542 29447 external_mini_cluster.cc:1428] Setting key dbc0568965e7aaf8a8337594c4039355
I20260810 15:00:21.806339 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.799030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.28.193.254:40367)
0810 15:00:21.799101 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.799104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.799267 (+ 163us) server_negotiation.cc:407] Connection header received
0810 15:00:21.799748 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.799749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.799764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.799817 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.800436 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.800923 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.801892 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.802094 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.802150 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.803999 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.804000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.804000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.804007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.805063 (+ 1056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.805447 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.805448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.805448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.805502 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.805745 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.805746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.805746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.805860 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.805918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.806165 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.806250 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:00:21.806485 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.798955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40367 (local address 127.0.0.1:33666)
0810 15:00:21.799150 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.799186 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.799646 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.799890 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.799893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.800141 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.800332 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.800336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.801009 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.801010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.801800 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.801802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.802166 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.802167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.802282 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.802644 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.802652 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.803910 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.805134 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.805141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.805143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.805354 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.805558 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.805559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.805559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.805668 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.805920 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.805927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.806052 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.806137 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.806282 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260810 15:00:21.807143 18857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.252@KRBTEST.COM; defaulting to no policy
I20260810 15:00:21.813663 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.807384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47152)
0810 15:00:21.807429 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.807438 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.807473 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.807690 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.807690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.807817 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.807859 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.807860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.808651 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.808652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.809123 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.809124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.809537 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.809538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.809596 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.809951 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.809955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.811010 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.812748 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.812749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.812749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.812919 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.813116 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.813116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.813116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.813159 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.813443 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.813444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.813480 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.813528 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.813600 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:00:21.813807 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.807425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.28.193.253:38081)
0810 15:00:21.807579 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.807582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.807596 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:00:21.807612 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.807614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.807623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.807682 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.807922 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.808573 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.809189 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.809477 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.809533 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.811090 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.811098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.811099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.811105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.812665 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.812996 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.812999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.813000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.813055 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.813211 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.813212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.813212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.813375 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.813455 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.813560 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.813678 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260810 15:00:21.814312 18857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:00:21.814604 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.814507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53094)
0810 15:00:21.814545 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.814586 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:00:21.814734 18761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
I20260810 15:00:21.815104 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.815015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53108)
0810 15:00:21.815053 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.815081 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:00:21.815217 18857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:36931: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
I20260810 15:00:21.830869 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:36931
--webserver_interface=127.28.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:21.864441 18857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } attempt: 1
I20260810 15:00:21.865001 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.864842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53120)
0810 15:00:21.864915 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.864961 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:00:21.865370 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.865257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53126)
0810 15:00:21.865314 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.865346 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 15:00:21.865466 18857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:36931: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
I20260810 15:00:21.869912 18723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } attempt: 3
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:00:21.877219 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.870234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47166)
0810 15:00:21.870289 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.870301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.870365 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.870490 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.870491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.870643 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.870710 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.870712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.871641 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.871642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.872165 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.872167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.872636 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.872637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.872702 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.873215 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.873222 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.874678 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.876317 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.876319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.876319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.876489 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.876674 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.876674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.876675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.876724 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.876983 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.876984 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.877025 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.877075 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.877150 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:00:21.877274 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.870293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.28.193.253:38081)
0810 15:00:21.870365 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.870368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.870379 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:21.870411 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.870412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.870424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.870484 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.870770 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.871566 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.872237 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.872559 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.872624 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.874775 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.874777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.874777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.874785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.876221 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.876560 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.876561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.876561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.876619 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.876792 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.876793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.876793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.876920 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.876987 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.877111 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.877189 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:00:21.877801 18723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:00:21.878144 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.878033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53140)
0810 15:00:21.878078 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.878114 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 15:00:21.878506 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.878412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53150)
0810 15:00:21.878451 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.878488 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 15:00:21.878594 18723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:36931: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
I20260810 15:00:21.909780 18723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } attempt: 1
I20260810 15:00:21.910254 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.910101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53160)
0810 15:00:21.910170 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.910217 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:00:21.910658 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.910556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53174)
0810 15:00:21.910601 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.910633 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:00:21.910806 18723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:36931: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
W20260810 15:00:21.916765 18866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:21.916954 18866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:21.916983 18866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:21.918110 18866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:21.918159 18866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:21.918177 18866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:21.918193 18866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:21.918208 18866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:21.919651 18866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:40367,127.28.193.253:38081,127.28.193.252:36931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:21.919838 18866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:21.920012 18866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:21.924443 18885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:21.924435 18887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:21.924432 18883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:21.924610 18866 server_base.cc:1034] running on GCE node
I20260810 15:00:21.924949 18866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:21.925230 18866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:21.926365 18866 hybrid_clock.cc:648] HybridClock initialized: now 1786374021926349 us; error 28 us; skew 500 ppm
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:21.928719 18866 init.cc:377] Logged in from keytab as kudu/127.28.193.252@KRBTEST.COM (short username kudu)
I20260810 15:00:21.929134 18866 webserver.cc:460] Webserver started at http://127.28.193.252:45005/ using document root <none> and password file <none>
I20260810 15:00:21.929330 18866 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:21.929378 18866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:21.929502 18866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:21.930541 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data/instance:
uuid: "7a91e186702147879a8dbee77ba957fe"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "e927baa8763a9349ebabc62d8a15de33"
server_key_iv: "72bb408c4b1cefebfe3eddbc581b7a63"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.930917 18866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal/instance:
uuid: "7a91e186702147879a8dbee77ba957fe"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "e927baa8763a9349ebabc62d8a15de33"
server_key_iv: "72bb408c4b1cefebfe3eddbc581b7a63"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.932205 18866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:00:21.932873 18899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:21.933027 18866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:21.933110 18866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "7a91e186702147879a8dbee77ba957fe"
format_stamp: "Formatted at 2026-08-10 15:00:21 on dist-test-slave-cvwc"
server_key: "e927baa8763a9349ebabc62d8a15de33"
server_key_iv: "72bb408c4b1cefebfe3eddbc581b7a63"
server_key_version: "encryptionkey@0"
I20260810 15:00:21.933176 18866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:21.933614 18857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } attempt: 2
I20260810 15:00:21.933981 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.933842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53184)
0810 15:00:21.933903 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.933944 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:00:21.934264 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.934186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53190)
0810 15:00:21.934211 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.934242 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260810 15:00:21.934362 18857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:36931: Network error: Client connection negotiation failed: client connection to 127.28.193.252:36931: connect: Connection refused (error 111)
I20260810 15:00:21.959095 18866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:21.960872 18866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:21.960997 18866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:21.964920 18866 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.252:36931
I20260810 15:00:21.964956 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.252:36931 every 8 connection(s)
I20260810 15:00:21.965289 18866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
I20260810 15:00:21.966073 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 18866
I20260810 15:00:21.966193 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786373869351853-29447-0/minicluster-data/master-2/wal/instance
I20260810 15:00:21.966316 18978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:21.966413 29447 external_mini_cluster.cc:1428] Setting key c30d90825c10b963c181ec07a03ff419
I20260810 15:00:21.968659 18978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:21.976644 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.969306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.28.193.254:40367)
0810 15:00:21.969361 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.969363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.969517 (+ 154us) server_negotiation.cc:407] Connection header received
0810 15:00:21.969574 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.969576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.969586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.969634 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.970211 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.970762 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.971676 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.971866 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.971912 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.974005 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.974007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.974007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.974015 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.975543 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.975860 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.975861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.975862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.975922 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.976085 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.976086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.976086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.976197 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.976252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.976460 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.976546 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:00:21.976831 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.969269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:40367 (local address 127.0.0.1:33678)
0810 15:00:21.969428 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.969460 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.969523 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.969658 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.969661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.970035 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.970120 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.970122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.970897 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.970897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.971596 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.971599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.971940 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.971941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.972021 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.972417 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.972428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.973921 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.975613 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.975616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.975618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.975788 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.975978 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.975978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.975979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.976036 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.976251 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.976259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.976340 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.976413 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.976732 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":162,"thread_start_us":85,"threads_started":1}
I20260810 15:00:21.976831 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.966947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.28.193.252:36931)
0810 15:00:21.967114 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.967124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.967150 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:00:21.967612 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.967619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.967666 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.967783 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.968538 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.969663 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.970525 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.971083 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.971158 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.973481 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.973489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.973490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.973497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.975331 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.975674 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.975677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.975678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.975766 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.975943 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.975944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.975945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.976177 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.976301 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.976412 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.976557 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260810 15:00:21.977387 18978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:21.983628 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.977620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:38081 (local address 127.0.0.1:47170)
0810 15:00:21.977740 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.977748 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.977790 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.977911 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.977911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.978023 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.978054 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.978055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.978727 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.978728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.979242 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.979245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.979616 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.979617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.979670 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.980032 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.980037 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.981207 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.982731 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.982732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.982732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.982961 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.983134 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.983134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.983135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.983196 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.983395 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.983396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.983439 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.983498 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.983566 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 15:00:21.983691 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.977679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.28.193.253:38081)
0810 15:00:21.977750 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.977752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.977779 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:00:21.977832 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.977833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.977843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.977908 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.978115 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.978654 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.979317 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.979558 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.979606 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.981287 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.981288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.981288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.981295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.982638 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.983027 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.983028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.983028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.983076 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.983249 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.983250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.983250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.983340 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.983389 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.983534 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.983604 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:00:21.984395 18978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:00:21.984612 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.977629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.28.193.254:40367)
0810 15:00:21.977662 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.977664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.977670 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:00:21.977688 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.977688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.977696 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.977776 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.978307 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.978762 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.979413 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.979582 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.979640 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.981917 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.981918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.981918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.981925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.983351 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.983682 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.983683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.983683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.983746 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.983945 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.983946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.983946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.984066 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.984226 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.984410 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.984524 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:00:21.985945 18723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } attempt: 2
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
Aug 10 15:00:21 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:00:21.992194 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.984670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.28.193.252:36931)
0810 15:00:21.984698 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.984700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.984708 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:00:21.984760 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.984761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.984768 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.984811 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.985021 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.985564 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.986209 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.986476 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.986547 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.989078 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.989079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.989079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.989085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.990537 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.991112 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.991113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.991114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.991175 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.991366 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.991367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.991368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.991481 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.991551 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.991706 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.992091 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":311,"server-negotiator.queue_time_us":16}
I20260810 15:00:21.992604 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.985477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.28.193.253:38081)
0810 15:00:21.985511 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.985513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.985521 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:00:21.985562 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.985563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.985570 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.985615 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.986159 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.986782 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.987654 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.987851 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.987914 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.990110 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.990112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.990112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.990118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.991535 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.991894 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.991895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.991895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.991952 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.992140 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.992142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.992142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.992246 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.992420 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.992455 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.992519 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:00:21.992700 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.984641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53200)
0810 15:00:21.984679 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.984688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.984720 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.984812 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.984813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.984937 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.984960 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.984961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.985623 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.985624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.986132 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.986135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.986547 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.986548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.986600 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.987692 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.987697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.989007 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.990622 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.990623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.990623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.991037 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.991244 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.991244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.991245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.991292 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.991559 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.991560 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.991604 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.991673 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.992031 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 15:00:21.993813 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.986206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53212)
0810 15:00:21.986248 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:21.986258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:21.986292 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:21.986625 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:21.986626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:21.986785 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:21.986822 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.986824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.988007 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.988008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.988732 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:21.988736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.989232 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.989233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:21.989290 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.989668 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:21.989672 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:21.990912 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:21.992652 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.992653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.992653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.992859 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.993187 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:21.993188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:21.993189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:21.993247 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:21.993541 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:21.993542 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:21.993582 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 15:00:21.993623 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 15:00:21.993704 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:00:21.994138 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.986361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.28.193.252:36931)
0810 15:00:21.986522 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.986523 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.986530 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:00:21.986547 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.986548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.986556 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.986618 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.986885 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.987935 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.988858 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.989148 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.989393 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:21.990989 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:21.990990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:21.990990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:21.990996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:21.992573 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.993053 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.993054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.993055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.993114 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:21.993311 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:21.993312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:21.993312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:21.993462 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:21.993926 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:21.993966 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 15:00:21.994033 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260810 15:00:21.994196 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:00:21.994367 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: Bootstrap starting.
I20260810 15:00:21.995042 18978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:21.995332 18978 log.cc:824] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:21.996277 18978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: No bootstrap required, opened a new log
I20260810 15:00:21.996879 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: Bootstrap starting.
I20260810 15:00:21.997422 18723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:21.997758 18723 log.cc:824] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:21.998163 18978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:21.998323 18978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:21.998368 18978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a91e186702147879a8dbee77ba957fe, State: Initialized, Role: FOLLOWER
I20260810 15:00:21.998498 18978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:21.998814 18978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:22.000662 18723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: No bootstrap required, opened a new log
I20260810 15:00:22.002707 18723 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.002854 18723 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:22.002892 18723 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcccbbc8b2c54a148d6e8776a6963b75, State: Initialized, Role: FOLLOWER
I20260810 15:00:22.003010 18723 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.003278 18723 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:22.003404 19029 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.003502 19029 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:00:22.002043 19025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:00:22.004621 19025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:00:22.002216 19007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.004715 19007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:00:22.004737 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.995542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.28.193.253:38081)
0810 15:00:21.995577 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.995579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.997793 (+ 2214us) server_negotiation.cc:407] Connection header received
0810 15:00:21.997957 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.997958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.997966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.997992 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.998990 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.999636 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.000615 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.000873 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.000913 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:22.002307 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:22.002308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:22.002309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:22.002315 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:22.003802 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.004092 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.004093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.004093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.004143 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.004299 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.004300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.004300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.004407 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:22.004499 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:22.004586 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 15:00:22.004655 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:00:22.002806 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.995492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.28.193.252:36931)
0810 15:00:21.995524 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.995526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.995837 (+ 311us) server_negotiation.cc:407] Connection header received
0810 15:00:21.995904 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.995905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.995915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.995969 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.996598 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.997340 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.998317 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.998631 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:21.998675 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:22.000028 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:22.000030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:22.000030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:22.000037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:22.001527 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.001986 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.001988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.001988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.002047 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.002244 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.002245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.002245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.002358 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:22.002460 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:22.002657 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 15:00:22.002735 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 15:00:22.006083 19048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:00:22.006182 19048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:00:22.006335 18607 master_runner.cc:186] Error getting master registration for 127.28.193.253:38081: OK, instance_id { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" instance_seqno: 1786374021794521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:00:22.008400 18607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:00:22.008626 18866 master_runner.cc:186] Error getting master registration for 127.28.193.253:38081: OK, instance_id { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" instance_seqno: 1786374021794521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:00:22.010244 18866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:00:22.017038 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:21.995489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.28.193.254:40367)
0810 15:00:21.995525 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:21.995526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:21.998036 (+ 2510us) server_negotiation.cc:407] Connection header received
0810 15:00:21.998121 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:21.998122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:21.998130 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:21.998155 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:21.998775 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:21.999411 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.004950 (+ 5539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.005204 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.005250 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:22.014723 (+ 9473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:22.014725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:22.014725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:22.014732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:22.016029 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.016340 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.016341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.016342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.016393 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.016559 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.016560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.016560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.016663 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:22.016759 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:22.016876 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 15:00:22.016955 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:00:22.025494 18857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } attempt: 3
Aug 10 15:00:22 dist-test-slave-cvwc krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:00:22.031629 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:22.025734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:36931 (local address 127.0.0.1:53232)
0810 15:00:22.025835 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:22.025844 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:22.025888 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:22.025993 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:22.025994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:22.026134 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:22.026174 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:22.026175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.026777 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.026778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:22.027259 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:22.027260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.027565 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.027566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:22.027621 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:22.027995 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:22.028000 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:22.029125 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:22.030655 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:22.030656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:22.030657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:22.030816 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:22.031006 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:22.031007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:22.031007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:22.031065 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:22.031303 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:22.031304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:22.031348 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 15:00:22.031429 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:00:22.031532 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:00:22.031629 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:22.025766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.28.193.252:36931)
0810 15:00:22.025805 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:22.025807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:22.025873 (+ 66us) server_negotiation.cc:407] Connection header received
0810 15:00:22.025930 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:22.025931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:22.025940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:22.025985 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:22.026236 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.026711 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.027328 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:22.027509 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:22.027562 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:22.029203 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:22.029204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:22.029204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:22.029211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:22.030592 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.030899 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.030900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.030900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.030954 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:22.031114 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:22.031114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:22.031115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:22.031247 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:22.031302 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:22.031454 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 15:00:22.031535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:00:22.033529 18857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: Bootstrap starting.
I20260810 15:00:22.034192 18857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:22.034469 18857 log.cc:824] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:22.035171 18857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: No bootstrap required, opened a new log
I20260810 15:00:22.036717 18857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.036839 18857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:22.036890 18857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d282bb4ab346458b0f75bf8a94adf7, State: Initialized, Role: FOLLOWER
I20260810 15:00:22.037009 18857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.037243 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.037309 18857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:22.037381 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:00:22.039129 19071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:00:22.039230 19071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:00:22.041050 18734 master_runner.cc:263] No leader master found from master 74d282bb4ab346458b0f75bf8a94adf7
I20260810 15:00:22.041093 18734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 15:00:22.042447 18767 tablet.cc:2369] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:22.142450 18643 tablet.cc:2369] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:22.211623 18906 tablet.cc:2369] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:22.280172 19029 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:22.280315 19029 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.280565 19029 leader_election.cc:288] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d282bb4ab346458b0f75bf8a94adf7 (127.28.193.253:38081), 7a91e186702147879a8dbee77ba957fe (127.28.193.252:36931)
I20260810 15:00:22.280857 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91e186702147879a8dbee77ba957fe" is_pre_election: true
I20260810 15:00:22.280857 18807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d282bb4ab346458b0f75bf8a94adf7" is_pre_election: true
I20260810 15:00:22.281028 18937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcccbbc8b2c54a148d6e8776a6963b75 in term 0.
I20260810 15:00:22.281026 18807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcccbbc8b2c54a148d6e8776a6963b75 in term 0.
I20260810 15:00:22.281246 18632 leader_election.cc:302] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91e186702147879a8dbee77ba957fe, fcccbbc8b2c54a148d6e8776a6963b75; no voters:
I20260810 15:00:22.281448 19029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:00:22.281510 19029 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:00:22.281525 19029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:22.282318 19029 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.282438 19029 leader_election.cc:288] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [CANDIDATE]: Term 1 election: Requested vote from peers 74d282bb4ab346458b0f75bf8a94adf7 (127.28.193.253:38081), 7a91e186702147879a8dbee77ba957fe (127.28.193.252:36931)
I20260810 15:00:22.282579 18807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d282bb4ab346458b0f75bf8a94adf7"
I20260810 15:00:22.282579 18937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a91e186702147879a8dbee77ba957fe"
I20260810 15:00:22.282660 18937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:22.282660 18807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:22.283478 18937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcccbbc8b2c54a148d6e8776a6963b75 in term 1.
I20260810 15:00:22.283478 18807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcccbbc8b2c54a148d6e8776a6963b75 in term 1.
I20260810 15:00:22.283641 18632 leader_election.cc:302] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a91e186702147879a8dbee77ba957fe, fcccbbc8b2c54a148d6e8776a6963b75; no voters:
I20260810 15:00:22.283771 19029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:22.283900 19029 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [term 1 LEADER]: Becoming Leader. State: Replica: fcccbbc8b2c54a148d6e8776a6963b75, State: Running, Role: LEADER
I20260810 15:00:22.283970 19029 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } }
I20260810 15:00:22.284327 19073 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcccbbc8b2c54a148d6e8776a6963b75. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.284401 19073 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:22.284593 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:22.284760 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:22.286044 18807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [term 1 FOLLOWER]: Refusing update from remote peer fcccbbc8b2c54a148d6e8776a6963b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:22.286054 18937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [term 1 FOLLOWER]: Refusing update from remote peer fcccbbc8b2c54a148d6e8776a6963b75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:22.286337 19073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:22.286458 19029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:22.287614 19007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcccbbc8b2c54a148d6e8776a6963b75. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.287616 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcccbbc8b2c54a148d6e8776a6963b75. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.287695 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:00:22.287695 19007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:00:22.288425 19029 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.288503 19029 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:22.288539 19029 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.288574 19029 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:22.288558 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.288615 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:00:22.288612 19079 mvcc.cc:205] Tried to move back new op lower bound from 7316987995281592320 to 7316987995275587584. Current Snapshot: MvccSnapshot[applied={T|T < 7316987995281592320}]
I20260810 15:00:22.288655 19007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcccbbc8b2c54a148d6e8776a6963b75" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 40367 } } peers { permanent_uuid: "74d282bb4ab346458b0f75bf8a94adf7" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 38081 } } peers { permanent_uuid: "7a91e186702147879a8dbee77ba957fe" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 36931 } } }
I20260810 15:00:22.288733 19007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:00:22.288718 19082 mvcc.cc:205] Tried to move back new op lower bound from 7316987995281592320 to 7316987995275587584. Current Snapshot: MvccSnapshot[applied={T|T < 7316987995281592320}]
I20260810 15:00:22.288889 19081 mvcc.cc:205] Tried to move back new op lower bound from 7316987995281592320 to 7316987995275587584. Current Snapshot: MvccSnapshot[applied={T|T < 7316987995281592320}]
I20260810 15:00:22.289590 19077 catalog_manager.cc:1347] Generated new cluster ID: 81170532eb32435387de44fe36d8f929
I20260810 15:00:22.289642 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:00:22.296033 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:00:22.296608 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:22.312211 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcccbbc8b2c54a148d6e8776a6963b75: Generated new TSK 0
I20260810 15:00:22.312420 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:00:23.005087 19025 catalog_manager.cc:1259] Loaded cluster ID: 81170532eb32435387de44fe36d8f929
I20260810 15:00:23.005169 19025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: loading cluster ID for follower catalog manager: success
I20260810 15:00:23.006055 19025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: acquiring CA information for follower catalog manager: success
I20260810 15:00:23.006359 19025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a91e186702147879a8dbee77ba957fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:00:23.039688 19071 catalog_manager.cc:1259] Loaded cluster ID: 81170532eb32435387de44fe36d8f929
I20260810 15:00:23.039765 19071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: loading cluster ID for follower catalog manager: success
I20260810 15:00:23.040571 19071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: acquiring CA information for follower catalog manager: success
I20260810 15:00:23.040890 19071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 74d282bb4ab346458b0f75bf8a94adf7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:00:23.370108 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 18607
I20260810 15:00:23.385636 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 18734
I20260810 15:00:23.397307 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 18866
2026-08-10T15:00:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3158 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:00:23 dist-test-slave-cvwc krb5kdc[19107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:00:23 dist-test-slave-cvwc krb5kdc[19107](info): set up 2 sockets
Aug 10 15:00:23 dist-test-slave-cvwc krb5kdc[19107](info): commencing operation
krb5kdc: starting...
W20260810 15:00:24.572077 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:00:24.694414 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:39355
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:35981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:39355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:24.775308 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:24.775524 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:24.775544 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:24.776654 19124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:24.776691 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:24.776705 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:24.776715 19124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:24.776726 19124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:24.778201 19124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:35981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:39355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:24.778409 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:24.778606 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:24.783377 19132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:24.783429 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:24.783442 19142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:24.783689 19124 server_base.cc:1034] running on GCE node
I20260810 15:00:24.783952 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:24.784237 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:24.785390 19124 hybrid_clock.cc:648] HybridClock initialized: now 1786374024785374 us; error 27 us; skew 500 ppm
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374024, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:24.787709 19124 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:24.788132 19124 webserver.cc:460] Webserver started at http://127.28.193.254:40211/ using document root <none> and password file <none>
I20260810 15:00:24.788334 19124 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:24.788383 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:24.788507 19124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:24.789549 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6"
format_stamp: "Formatted at 2026-08-10 15:00:24 on dist-test-slave-cvwc"
server_key: "28240360ef72153b28d959fa9743c6c1"
server_key_iv: "cb7a714361bff4022a51f7954f472fd1"
server_key_version: "encryptionkey@0"
I20260810 15:00:24.789915 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6"
format_stamp: "Formatted at 2026-08-10 15:00:24 on dist-test-slave-cvwc"
server_key: "28240360ef72153b28d959fa9743c6c1"
server_key_iv: "cb7a714361bff4022a51f7954f472fd1"
server_key_version: "encryptionkey@0"
I20260810 15:00:24.791150 19124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:00:24.791805 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:24.791960 19124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:24.792044 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6"
format_stamp: "Formatted at 2026-08-10 15:00:24 on dist-test-slave-cvwc"
server_key: "28240360ef72153b28d959fa9743c6c1"
server_key_iv: "cb7a714361bff4022a51f7954f472fd1"
server_key_version: "encryptionkey@0"
I20260810 15:00:24.792104 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:24.806057 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:24.807727 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:24.807857 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:24.811691 19124 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:39355
I20260810 15:00:24.811724 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:39355 every 8 connection(s)
I20260810 15:00:24.812041 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:24.812887 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:24.815176 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6: Bootstrap starting.
I20260810 15:00:24.815711 19250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:24.815968 19250 log.cc:824] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:24.816602 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6: No bootstrap required, opened a new log
I20260810 15:00:24.818163 19250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } }
I20260810 15:00:24.818297 19250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:24.818331 19250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075af4d4c3ba4abbbe72c9bb4955ddc6, State: Initialized, Role: FOLLOWER
I20260810 15:00:24.818477 19250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } }
I20260810 15:00:24.818538 19250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:24.818569 19250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:24.818615 19250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:24.819087 19250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } }
I20260810 15:00:24.819191 19250 leader_election.cc:302] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 075af4d4c3ba4abbbe72c9bb4955ddc6; no voters:
I20260810 15:00:24.819357 19250 leader_election.cc:288] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:00:24.819367 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 19124
I20260810 15:00:24.819387 19253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:24.819455 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:00:24.819499 19253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [term 1 LEADER]: Becoming Leader. State: Replica: 075af4d4c3ba4abbbe72c9bb4955ddc6, State: Running, Role: LEADER
I20260810 15:00:24.819584 19253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } }
I20260810 15:00:24.819624 19250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:24.819671 29447 external_mini_cluster.cc:1428] Setting key 020e294ac5583f1102f373d0bd69eceb
I20260810 15:00:24.820338 19259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } } }
I20260810 15:00:24.820432 19259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:24.820585 19257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 075af4d4c3ba4abbbe72c9bb4955ddc6. Latest consensus state: current_term: 1 leader_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "075af4d4c3ba4abbbe72c9bb4955ddc6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 39355 } } }
I20260810 15:00:24.820657 19257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:24.820633 19276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:24.821228 19276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:24.823030 19276 catalog_manager.cc:1347] Generated new cluster ID: 0d33ff49690c48eabb1764fb07f2fae9
I20260810 15:00:24.823087 19276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:24.829912 19270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:24.820068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.28.193.254:39355)
0810 15:00:24.821230 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:24.821240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:24.821276 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:00:24.821409 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:24.821418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:24.821483 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:24.821652 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:24.822377 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:24.823444 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:24.824227 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:24.824651 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:24.824743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:24.826868 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:24.826873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:24.826873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:24.826890 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:24.828345 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:24.828668 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:24.828671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:24.828671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:24.828762 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:24.828938 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:24.828939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:24.828939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:24.829167 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:24.829395 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:24.829527 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 15:00:24.829671 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":63,"threads_started":1}
I20260810 15:00:24.830858 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:00:24.842756 19276 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:00:24.843464 19276 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:24.849375 19276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 075af4d4c3ba4abbbe72c9bb4955ddc6: Generated new TSK 0
I20260810 15:00:24.849587 19276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:00:24.884482 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 19124
2026-08-10T15:00:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1486 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19294](info): set up 2 sockets
Aug 10 15:00:24 dist-test-slave-cvwc krb5kdc[19294](info): commencing operation
krb5kdc: starting...
W20260810 15:00:26.049173 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:00:26.165015 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:26.246274 19314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:26.246479 19314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:26.246500 19314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:26.247614 19314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:26.247659 19314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:26.247673 19314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:26.247684 19314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:26.247697 19314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:26.249179 19314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:26.249405 19314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:26.249621 19314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:26.254213 19323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.254212 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.254225 19324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.254446 19314 server_base.cc:1034] running on GCE node
I20260810 15:00:26.254676 19314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:26.254915 19314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:26.256062 19314 hybrid_clock.cc:648] HybridClock initialized: now 1786374026256045 us; error 28 us; skew 500 ppm
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:26.261514 19314 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:26.261979 19314 webserver.cc:460] Webserver started at http://127.28.193.254:34159/ using document root <none> and password file <none>
I20260810 15:00:26.262179 19314 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:26.262226 19314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:26.262353 19314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:26.263458 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "8c879231144442458856d07885aabc79"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "b62e0f4e4013eb694a1d6ee3831a9adb"
server_key_iv: "6602c9bd4410f35de55ef8a6db53a8df"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.263796 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "8c879231144442458856d07885aabc79"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "b62e0f4e4013eb694a1d6ee3831a9adb"
server_key_iv: "6602c9bd4410f35de55ef8a6db53a8df"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.265053 19314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:00:26.265832 19339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.266031 19314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:26.266134 19314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "8c879231144442458856d07885aabc79"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "b62e0f4e4013eb694a1d6ee3831a9adb"
server_key_iv: "6602c9bd4410f35de55ef8a6db53a8df"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.266211 19314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:26.277221 19314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:26.279089 19314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:26.279223 19314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:26.283130 19314 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:32821
I20260810 15:00:26.283164 19437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:32821 every 8 connection(s)
I20260810 15:00:26.283506 19314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:26.284443 19439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:26.286820 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap starting.
I20260810 15:00:26.287398 19439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:26.287658 19439 log.cc:824] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:26.288311 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: No bootstrap required, opened a new log
I20260810 15:00:26.290014 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 19314
I20260810 15:00:26.289949 19439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:26.290133 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:00:26.290126 19439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:26.290165 19439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c879231144442458856d07885aabc79, State: Initialized, Role: FOLLOWER
I20260810 15:00:26.290352 29447 external_mini_cluster.cc:1428] Setting key 9c0425646a39c143603744c9a930b0f1
I20260810 15:00:26.290315 19439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:26.290395 19439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:26.290436 19439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:26.290482 19439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:26.291121 19439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:26.291218 19439 leader_election.cc:302] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c879231144442458856d07885aabc79; no voters:
I20260810 15:00:26.291378 19439 leader_election.cc:288] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:00:26.291414 19447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:26.291543 19447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 LEADER]: Becoming Leader. State: Replica: 8c879231144442458856d07885aabc79, State: Running, Role: LEADER
I20260810 15:00:26.291615 19439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:26.291631 19447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:26.293247 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:26.293223 19448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c879231144442458856d07885aabc79. Latest consensus state: current_term: 1 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:26.293376 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:26.293382 19448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:26.293596 19464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:26.294080 19464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:26.295668 19464 catalog_manager.cc:1347] Generated new cluster ID: 6a9b1437305847c1a63abf18bc64659c
I20260810 15:00:26.295723 19464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:26.299378 19464 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:00:26.299841 19464 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:26.301405 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.290813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.28.193.254:32821)
0810 15:00:26.291278 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.291289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.291323 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:00:26.291446 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.291453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.291512 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.292323 (+ 811us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:26.292399 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.293801 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.294556 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.295107 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.295224 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.297902 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:26.297906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:26.297907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:26.297919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:26.299843 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.300267 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.300271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.300271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.300376 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.300588 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.300588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.300589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.300835 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:26.300962 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.301087 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.301219 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":356,"threads_started":1}
I20260810 15:00:26.311595 19464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Generated new TSK 0
I20260810 15:00:26.311755 19464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:00:26.340627 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:26.423135 19478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:26.423339 19478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:26.423362 19478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:26.424422 19478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:26.424455 19478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:26.424479 19478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:26.425910 19478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:26.426117 19478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:26.426294 19478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:26.431043 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.431110 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.431185 19478 server_base.cc:1034] running on GCE node
W20260810 15:00:26.431272 19488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.431473 19478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:26.431684 19478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:26.432824 19478 hybrid_clock.cc:648] HybridClock initialized: now 1786374026432812 us; error 29 us; skew 500 ppm
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:26.435122 19478 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:26.435521 19478 webserver.cc:460] Webserver started at http://127.28.193.193:38019/ using document root <none> and password file <none>
I20260810 15:00:26.435755 19478 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:26.435802 19478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:26.435925 19478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:26.436924 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "9dcf8ffcce4c461baae4f234b01b657f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "e2940e8cf0684e6e37e7efe078bc7b36"
server_key_iv: "2b93b698e7b0b68c1cd6dd3daca1f3a1"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.437273 19478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "9dcf8ffcce4c461baae4f234b01b657f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "e2940e8cf0684e6e37e7efe078bc7b36"
server_key_iv: "2b93b698e7b0b68c1cd6dd3daca1f3a1"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.438596 19478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:00:26.439242 19506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.439388 19478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:26.439476 19478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "9dcf8ffcce4c461baae4f234b01b657f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "e2940e8cf0684e6e37e7efe078bc7b36"
server_key_iv: "2b93b698e7b0b68c1cd6dd3daca1f3a1"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.439534 19478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:26.454967 19478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:26.456653 19478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:26.456763 19478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:26.456926 19478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:26.457175 19478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:26.457203 19478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.457222 19478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:26.457239 19478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.462888 19478 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:36127
I20260810 15:00:26.462955 19687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:36127 every 8 connection(s)
I20260810 15:00:26.463241 19478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:00:26.466138 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 19478
I20260810 15:00:26.466259 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:00:26.466470 29447 external_mini_cluster.cc:1428] Setting key c8be24a6da4264441dcdc5ca5296511c
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:26.472615 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.463967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54331 (local address 127.28.193.254:32821)
0810 15:00:26.464053 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.464055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.464170 (+ 115us) server_negotiation.cc:407] Connection header received
0810 15:00:26.464647 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.464649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.464658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.464696 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:26.465561 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.466108 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.467436 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.467633 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.467690 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.469931 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:26.469932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:26.469932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:26.469939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:26.471164 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.471621 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.471622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.471623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.471690 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.472009 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.472010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.472010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.472126 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:26.472182 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.472486 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.472551 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:00:26.472864 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.463894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:54331)
0810 15:00:26.464060 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.464098 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.464559 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.464752 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.464755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.465302 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:26.465494 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.465499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.466191 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.466192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.467355 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.467358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.467713 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.467715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.467855 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.468336 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:26.468346 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:26.469839 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:26.471247 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.471255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.471257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.471546 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.471788 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.471789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.471789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.471930 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.472188 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:26.472193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:26.472355 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.472475 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.472637 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260810 15:00:26.473366 19688 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:26.473467 19688 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:26.473659 19688 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:26.474203 19370 ts_manager.cc:194] Registered new tserver with Master: 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127)
I20260810 15:00:26.474797 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:54331
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:00:26.492357 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:26.573107 19699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:26.573310 19699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:26.573335 19699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:26.574492 19699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:26.574534 19699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:26.574559 19699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:26.575997 19699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:26.576213 19699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:26.576426 19699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:26.581311 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.581321 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.581436 19699 server_base.cc:1034] running on GCE node
W20260810 15:00:26.581354 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.581748 19699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:26.582002 19699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:26.583160 19699 hybrid_clock.cc:648] HybridClock initialized: now 1786374026583145 us; error 29 us; skew 500 ppm
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:26.585520 19699 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:26.585973 19699 webserver.cc:460] Webserver started at http://127.28.193.194:46021/ using document root <none> and password file <none>
I20260810 15:00:26.586186 19699 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:26.586234 19699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:26.586356 19699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:26.587332 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d096bb99de4ba859bd54316d23605e14"
server_key_iv: "4edce5167aca84e3ddb6564da18cd4b4"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.587651 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d096bb99de4ba859bd54316d23605e14"
server_key_iv: "4edce5167aca84e3ddb6564da18cd4b4"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.588824 19699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:00:26.589454 19726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.589638 19699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:26.589769 19699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d096bb99de4ba859bd54316d23605e14"
server_key_iv: "4edce5167aca84e3ddb6564da18cd4b4"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.589835 19699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:26.603543 19699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:26.605537 19699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:26.605670 19699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:26.605906 19699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:26.606191 19699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:26.606225 19699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.606251 19699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:26.606268 19699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.611943 19699 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44087
I20260810 15:00:26.611999 19930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44087 every 8 connection(s)
I20260810 15:00:26.612298 19699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:00:26.617302 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 19699
I20260810 15:00:26.617400 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:00:26.617570 29447 external_mini_cluster.cc:1428] Setting key fabc91b3f4618273977e1b47094a743e
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:26.621261 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.612991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:37943 (local address 127.28.193.254:32821)
0810 15:00:26.613053 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.613056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.613226 (+ 170us) server_negotiation.cc:407] Connection header received
0810 15:00:26.613759 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.613760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.613774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.613825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:26.614752 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.615248 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.616194 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.616390 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.616455 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.618713 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:26.618714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:26.618715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:26.618722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:26.619942 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.620374 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.620375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.620376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.620445 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.620645 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.620645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.620646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.620778 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:26.620844 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.621083 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.621171 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:00:26.621333 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.612950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:37943)
0810 15:00:26.613112 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.613148 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.613645 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.613897 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.613900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.614464 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:26.614652 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.614657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.615332 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.615334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.616111 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.616113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.616457 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.616457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.616573 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.617007 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:26.617015 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:26.618600 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:26.620034 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.620044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.620047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.620305 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.620502 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.620503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.620503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.620595 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.620830 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:26.620833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:26.620966 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.621045 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.621164 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260810 15:00:26.621796 19938 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:26.621912 19938 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:26.622094 19938 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:26.622522 19370 ts_manager.cc:194] Registered new tserver with Master: fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087)
I20260810 15:00:26.623032 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:37943
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:00:26.644500 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:26.726647 19954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:26.726840 19954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:26.726871 19954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:26.727983 19954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:26.728024 19954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:26.728048 19954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:26.729485 19954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:26.729697 19954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:26.729943 19954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:26.734766 19971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.734762 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:26.734762 19966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:26.734881 19954 server_base.cc:1034] running on GCE node
I20260810 15:00:26.735191 19954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:26.735440 19954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:26.736569 19954 hybrid_clock.cc:648] HybridClock initialized: now 1786374026736556 us; error 33 us; skew 500 ppm
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:26.739004 19954 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:26.739439 19954 webserver.cc:460] Webserver started at http://127.28.193.195:42123/ using document root <none> and password file <none>
I20260810 15:00:26.739657 19954 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:26.739704 19954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:26.739825 19954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:26.740926 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "2705ded81f5f4823a1bd7a7460ce1135"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d0fc02faf40e81e43db0e9fe27ec6fd4"
server_key_iv: "1aebd6b6a7d4091b1e9e93f1fa43c97d"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.741272 19954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "2705ded81f5f4823a1bd7a7460ce1135"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d0fc02faf40e81e43db0e9fe27ec6fd4"
server_key_iv: "1aebd6b6a7d4091b1e9e93f1fa43c97d"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.742606 19954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:00:26.743271 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.743418 19954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:26.743494 19954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "2705ded81f5f4823a1bd7a7460ce1135"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d0fc02faf40e81e43db0e9fe27ec6fd4"
server_key_iv: "1aebd6b6a7d4091b1e9e93f1fa43c97d"
server_key_version: "encryptionkey@0"
I20260810 15:00:26.743553 19954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:26.751154 19954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:26.752954 19954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:26.753082 19954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:26.753280 19954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:26.753559 19954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:26.753592 19954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.753618 19954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:26.753644 19954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:26.759418 19954 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:39545
I20260810 15:00:26.759467 20176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:39545 every 8 connection(s)
I20260810 15:00:26.759785 19954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:00:26 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:26.767869 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.760502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:44523 (local address 127.28.193.254:32821)
0810 15:00:26.760567 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.760569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.760743 (+ 174us) server_negotiation.cc:407] Connection header received
0810 15:00:26.761224 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.761226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.761235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.761292 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:26.762211 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.762701 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.763624 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.763792 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.763834 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.765800 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:26.765802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:26.765802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:26.765809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:26.766855 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.767190 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.767191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.767191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.767234 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.767415 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.767416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.767416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.767511 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:26.767555 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.767749 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.767805 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:00:26.768059 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.760458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:44523)
0810 15:00:26.760632 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.760674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.761147 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.761347 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.761350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.761950 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:26.762142 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.762147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.762764 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.762764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.763564 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.763567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.763843 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.763843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.763932 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.764401 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:26.764410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:26.765677 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:26.766921 (+ 1244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.766926 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.766928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.767135 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.767277 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:26.767277 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:26.767277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:26.767363 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:26.767551 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:26.767553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:26.767661 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.767734 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.767849 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260810 15:00:26.768589 20178 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:26.768685 20178 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:26.768854 20178 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:26.769251 19370 ts_manager.cc:194] Registered new tserver with Master: 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
I20260810 15:00:26.769636 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:44523
I20260810 15:00:26.769776 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 19954
I20260810 15:00:26.769886 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:00:26.770069 29447 external_mini_cluster.cc:1428] Setting key fad628d0de24abce179ac3d40dc645fe
I20260810 15:00:26.770862 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:00:26.777148 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.772073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.28.193.254:32821)
0810 15:00:26.772124 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.772126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.772256 (+ 130us) server_negotiation.cc:407] Connection header received
0810 15:00:26.772296 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.772297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.772305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.772363 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:26.772850 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.773302 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.774048 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.774220 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.774266 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.775218 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:26.775219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:26.775219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:26.775224 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:26.776372 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.776622 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.776622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.776622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.776664 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:26.776767 (+ 103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:26.776767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:26.776767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:26.776885 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:26.776949 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.777037 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.777089 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:00:26.778880 19370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:00:26.780395 19370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:00:26.786079 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44087 (local address 127.0.0.1:57820)
0810 15:00:26.782519 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.782559 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.782626 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.782881 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.782885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.782887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:26.782951 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.782953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.783941 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.783942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.785018 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.785021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.785799 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.785800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.785875 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.785888 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.785954 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.786006 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260810 15:00:26.786091 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.28.193.194:44087)
0810 15:00:26.782704 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.782709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.782720 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:26.782761 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.782764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.782779 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.782868 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:26.783036 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.783852 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.785111 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.785735 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.785800 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.785860 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.785978 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.786015 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260810 15:00:26.786595 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36127 (local address 127.0.0.1:41380)
0810 15:00:26.782873 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.782883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.782990 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.783258 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.783258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.783260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:26.783306 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.783307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.784510 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.784511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.785328 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.785330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.786358 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.786359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.786425 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.786430 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.786483 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.786521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":22,"thread_start_us":92,"threads_started":1}
I20260810 15:00:26.786672 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.28.193.195:39545)
0810 15:00:26.782866 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.782869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.782878 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:00:26.782913 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.782914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.782929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.782998 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:26.783192 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.784272 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.785500 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.786197 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.786268 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.786321 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.786563 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.786603 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260810 15:00:26.786798 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39545 (local address 127.0.0.1:56136)
0810 15:00:26.782710 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:26.782718 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:26.782755 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:26.782999 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:26.783000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:26.783002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:26.783114 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.783117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.784374 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.784375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.785416 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:26.785419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.786357 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.786358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:26.786425 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.786431 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:00:26.786478 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:00:26.786521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":8,"thread_start_us":52,"threads_started":1}
I20260810 15:00:26.787007 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:26.782573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.28.193.193:36127)
0810 15:00:26.782752 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:26.782755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:26.782960 (+ 205us) server_negotiation.cc:407] Connection header received
0810 15:00:26.783085 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:26.783088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:26.783105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:26.783156 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:26.783365 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.784427 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.785403 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:26.786231 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:26.786287 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:26.786329 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:26.786513 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:00:26.786557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260810 15:00:26.787021 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 (DEFAULT_TABLE table=test-table [id=f6afbc10904641998126216993cd019a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:26.787392 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ebecb089e2417886640dcf1b2dfc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:26.787454 20057 tablet_service.cc:1467] Processing CreateTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 (DEFAULT_TABLE table=test-table [id=f6afbc10904641998126216993cd019a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:26.787765 20057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ebecb089e2417886640dcf1b2dfc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:26.790015 20196 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Bootstrap starting.
I20260810 15:00:26.790534 19583 tablet_service.cc:1467] Processing CreateTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 (DEFAULT_TABLE table=test-table [id=f6afbc10904641998126216993cd019a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:26.790699 20196 tablet_bootstrap.cc:654] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:26.790817 19583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ebecb089e2417886640dcf1b2dfc60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:26.791158 20196 log.cc:824] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:26.791527 20199 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Bootstrap starting.
I20260810 15:00:26.791862 20196 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: No bootstrap required, opened a new log
I20260810 15:00:26.791918 20196 ts_tablet_manager.cc:1397] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:26.793370 20201 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap starting.
I20260810 15:00:26.793354 20196 raft_consensus.cc:348] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.793542 20196 raft_consensus.cc:374] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:26.793588 20196 raft_consensus.cc:729] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe85dd5d4de4477cafa3aca9e320d24f, State: Initialized, Role: FOLLOWER
I20260810 15:00:26.793730 20196 consensus_queue.cc:260] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.794011 20196 ts_tablet_manager.cc:1428] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:26.794143 19938 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
I20260810 15:00:26.794157 20201 tablet_bootstrap.cc:654] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:26.794703 20199 tablet_bootstrap.cc:654] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:26.794706 20201 log.cc:824] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:26.795161 20199 log.cc:824] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:26.795469 20201 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: No bootstrap required, opened a new log
I20260810 15:00:26.795529 20201 ts_tablet_manager.cc:1397] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:26.795795 20199 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: No bootstrap required, opened a new log
I20260810 15:00:26.795848 20199 ts_tablet_manager.cc:1397] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:00:26.796829 20201 raft_consensus.cc:348] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.796957 20201 raft_consensus.cc:374] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:26.796988 20201 raft_consensus.cc:729] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcf8ffcce4c461baae4f234b01b657f, State: Initialized, Role: FOLLOWER
I20260810 15:00:26.797094 20201 consensus_queue.cc:260] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.797153 20199 raft_consensus.cc:348] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.797303 20199 raft_consensus.cc:374] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:26.797340 20199 raft_consensus.cc:729] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2705ded81f5f4823a1bd7a7460ce1135, State: Initialized, Role: FOLLOWER
I20260810 15:00:26.797341 20201 ts_tablet_manager.cc:1428] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:26.797435 19688 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
I20260810 15:00:26.797446 20199 consensus_queue.cc:260] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:26.797698 20178 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
I20260810 15:00:26.797685 20199 ts_tablet_manager.cc:1428] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260810 15:00:26.862847 19946 tablet.cc:2369] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:26.963927 19690 tablet.cc:2369] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:27.010373 20179 tablet.cc:2369] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:27.217226 20209 raft_consensus.cc:482] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:27.217384 20209 raft_consensus.cc:504] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:27.217811 20209 leader_election.cc:288] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
I20260810 15:00:27.220721 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.218121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:55445 (local address 127.28.193.195:39545)
0810 15:00:27.218170 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.218172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.218211 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:00:27.218297 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.218298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.218306 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.218359 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:27.218527 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.218985 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.219926 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.220396 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.220462 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.220472 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.220619 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.220651 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:00:27.220721 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.218059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33933 (local address 127.28.193.194:44087)
0810 15:00:27.218111 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.218113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.218215 (+ 102us) server_negotiation.cc:407] Connection header received
0810 15:00:27.218266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.218267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.218277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.218324 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:27.218518 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.218989 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.219929 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.220360 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.220421 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.220430 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.220608 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.220650 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:00:27.220734 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.217999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39545 (local address 127.28.193.193:55445)
0810 15:00:27.218149 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.218160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.218225 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.218369 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.218370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.218372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:27.218449 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.218451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.219056 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.219057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.219835 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.219837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.220495 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.220495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.220553 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.220555 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.220605 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.220634 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260810 15:00:27.220734 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.217994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44087 (local address 127.28.193.193:33933)
0810 15:00:27.218126 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.218137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.218212 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.218344 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.218345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.218348 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:27.218434 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.218436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.219065 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.219066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.219841 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.219843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.220464 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.220465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.220533 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.220537 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.220592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.220630 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":10,"thread_start_us":71,"threads_started":1}
I20260810 15:00:27.221103 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ebecb089e2417886640dcf1b2dfc60" candidate_uuid: "9dcf8ffcce4c461baae4f234b01b657f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705ded81f5f4823a1bd7a7460ce1135" is_pre_election: true
I20260810 15:00:27.221177 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ebecb089e2417886640dcf1b2dfc60" candidate_uuid: "9dcf8ffcce4c461baae4f234b01b657f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" is_pre_election: true
I20260810 15:00:27.221269 20090 raft_consensus.cc:2454] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dcf8ffcce4c461baae4f234b01b657f in term 0.
I20260810 15:00:27.221333 19853 raft_consensus.cc:2454] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dcf8ffcce4c461baae4f234b01b657f in term 0.
I20260810 15:00:27.221493 19509 leader_election.cc:302] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2705ded81f5f4823a1bd7a7460ce1135, 9dcf8ffcce4c461baae4f234b01b657f; no voters:
I20260810 15:00:27.221626 20209 raft_consensus.cc:2781] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:00:27.221683 20209 raft_consensus.cc:482] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:00:27.221707 20209 raft_consensus.cc:3037] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:27.222432 20209 raft_consensus.cc:504] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:27.222550 20209 leader_election.cc:288] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 1 election: Requested vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
I20260810 15:00:27.222703 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ebecb089e2417886640dcf1b2dfc60" candidate_uuid: "9dcf8ffcce4c461baae4f234b01b657f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705ded81f5f4823a1bd7a7460ce1135"
I20260810 15:00:27.222704 19853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ebecb089e2417886640dcf1b2dfc60" candidate_uuid: "9dcf8ffcce4c461baae4f234b01b657f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
I20260810 15:00:27.222772 20090 raft_consensus.cc:3037] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:27.222774 19853 raft_consensus.cc:3037] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:27.223469 20090 raft_consensus.cc:2454] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcf8ffcce4c461baae4f234b01b657f in term 1.
I20260810 15:00:27.223477 19853 raft_consensus.cc:2454] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dcf8ffcce4c461baae4f234b01b657f in term 1.
I20260810 15:00:27.223616 19509 leader_election.cc:302] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2705ded81f5f4823a1bd7a7460ce1135, 9dcf8ffcce4c461baae4f234b01b657f; no voters:
I20260810 15:00:27.223708 20209 raft_consensus.cc:2781] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:27.223839 20209 raft_consensus.cc:686] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 LEADER]: Becoming Leader. State: Replica: 9dcf8ffcce4c461baae4f234b01b657f, State: Running, Role: LEADER
I20260810 15:00:27.223940 20209 consensus_queue.cc:237] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:27.224776 19370 catalog_manager.cc:5495] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193). New cstate: current_term: 1 leader_uuid: "9dcf8ffcce4c461baae4f234b01b657f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } health_report { overall_health: UNKNOWN } } }
I20260810 15:00:27.334499 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.331396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.28.193.193:36127)
0810 15:00:27.331527 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.331529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.331541 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:00:27.331591 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.331592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.331617 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.331701 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:00:27.331854 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.332483 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.333467 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.333776 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.333844 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.334038 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:00:27.334180 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:00:27.334282 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.334308 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.334335 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260810 15:00:27.336159 20090 raft_consensus.cc:1261] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Refusing update from remote peer 9dcf8ffcce4c461baae4f234b01b657f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:27.336172 19853 raft_consensus.cc:1261] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Refusing update from remote peer 9dcf8ffcce4c461baae4f234b01b657f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:27.336496 20213 consensus_queue.cc:1035] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:27.336618 20209 consensus_queue.cc:1035] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:27.338871 20221 mvcc.cc:205] Tried to move back new op lower bound from 7316988015966187520 to 7316988015509786624. Current Snapshot: MvccSnapshot[applied={T|T < 7316988015966187520}]
I20260810 15:00:27.339354 20228 mvcc.cc:205] Tried to move back new op lower bound from 7316988015966187520 to 7316988015509786624. Current Snapshot: MvccSnapshot[applied={T|T < 7316988015966187520}]
I20260810 15:00:27.340137 20229 mvcc.cc:205] Tried to move back new op lower bound from 7316988015966187520 to 7316988015509786624. Current Snapshot: MvccSnapshot[applied={T|T < 7316988015966187520 or (T in {7316988015966187520})}]
I20260810 15:00:27.341498 19367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:27.341580 19367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:27.342284 19367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Sending DeleteTablet for 3 replicas of tablet 40ebecb089e2417886640dcf1b2dfc60
I20260810 15:00:27.342583 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:27 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:57820
I20260810 15:00:27.342597 19583 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:27 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41380
I20260810 15:00:27.342813 20240 tablet_replica.cc:330] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: stopping tablet replica
I20260810 15:00:27.342797 20057 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:27 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:56136
I20260810 15:00:27.342911 20240 raft_consensus.cc:2229] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:00:27.342906 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 19478
I20260810 15:00:27.343014 20240 raft_consensus.cc:2258] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:27.350183 20242 tablet_replica.cc:330] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: stopping tablet replica
I20260810 15:00:27.350281 20242 raft_consensus.cc:2229] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:27.350360 20242 raft_consensus.cc:2258] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:27.350517 20244 tablet_replica.cc:330] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: stopping tablet replica
I20260810 15:00:27.350611 20244 raft_consensus.cc:2229] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:27.350694 20244 raft_consensus.cc:2258] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:27.350880 20242 ts_tablet_manager.cc:1905] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:27.351219 20244 ts_tablet_manager.cc:1905] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:27.352620 20244 ts_tablet_manager.cc:1918] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:27.352684 20244 log.cc:1196] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/40ebecb089e2417886640dcf1b2dfc60
I20260810 15:00:27.352808 20242 ts_tablet_manager.cc:1918] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:27.352854 20244 ts_tablet_manager.cc:1939] T 40ebecb089e2417886640dcf1b2dfc60 P 2705ded81f5f4823a1bd7a7460ce1135: Deleting consensus metadata
I20260810 15:00:27.352891 20242 log.cc:1196] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/40ebecb089e2417886640dcf1b2dfc60
I20260810 15:00:27.353195 20242 ts_tablet_manager.cc:1939] T 40ebecb089e2417886640dcf1b2dfc60 P fe85dd5d4de4477cafa3aca9e320d24f: Deleting consensus metadata
I20260810 15:00:27.353245 19342 catalog_manager.cc:4841] TS 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): tablet 40ebecb089e2417886640dcf1b2dfc60 (table test-table [id=f6afbc10904641998126216993cd019a]) successfully deleted
I20260810 15:00:27.353699 19340 catalog_manager.cc:4841] TS fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): tablet 40ebecb089e2417886640dcf1b2dfc60 (table test-table [id=f6afbc10904641998126216993cd019a]) successfully deleted
W20260810 15:00:27.361974 19343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:36127 (error 108)
I20260810 15:00:27.362267 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 19699
I20260810 15:00:27.364137 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.362361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36127 (local address 127.0.0.1:41386)
0810 15:00:27.364027 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.364090 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":39,"threads_started":1}
W20260810 15:00:27.365979 19343 catalog_manager.cc:4584] TS 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40ebecb089e2417886640dcf1b2dfc60: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36127: connect: Connection refused (error 111)
I20260810 15:00:27.370839 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 19954
I20260810 15:00:27.380553 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 19314
I20260810 15:00:27.387152 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--webserver_interface=127.28.193.254
--webserver_port=34159
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:27.467655 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:27.467857 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:27.467877 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:27.469080 20251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:27.469130 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:27.469151 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:27.469168 20251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:27.469185 20251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:27.470798 20251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=34159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:27.471016 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:27.471203 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:27.482118 20276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.482170 20271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.482182 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.482491 20251 server_base.cc:1034] running on GCE node
I20260810 15:00:27.482775 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:27.483122 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:27.484277 20251 hybrid_clock.cc:648] HybridClock initialized: now 1786374027484260 us; error 29 us; skew 500 ppm
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:27.486899 20251 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:27.487397 20251 webserver.cc:460] Webserver started at http://127.28.193.254:34159/ using document root <none> and password file <none>
I20260810 15:00:27.487632 20251 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:27.487699 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:27.489078 20251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:27.489693 20285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.489857 20251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:27.489948 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "8c879231144442458856d07885aabc79"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "b62e0f4e4013eb694a1d6ee3831a9adb"
server_key_iv: "6602c9bd4410f35de55ef8a6db53a8df"
server_key_version: "encryptionkey@0"
I20260810 15:00:27.490356 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:27.509351 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:27.511126 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:27.511274 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:27.515331 20251 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:32821
I20260810 15:00:27.515444 20373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:32821 every 8 connection(s)
I20260810 15:00:27.515753 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:27.517496 20374 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:00:27.518234 20374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap starting.
I20260810 15:00:27.522266 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 20251
I20260810 15:00:27.522624 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:36127
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=38019
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:27.527129 20374 log.cc:824] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:27.529138 20374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:00:27.529443 20374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap complete.
I20260810 15:00:27.531679 20374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:27.531922 20374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c879231144442458856d07885aabc79, State: Initialized, Role: FOLLOWER
I20260810 15:00:27.532068 20374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:27.532136 20374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:27.532183 20374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:27.532231 20374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:00:27.533246 20374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:27.533340 20374 leader_election.cc:302] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c879231144442458856d07885aabc79; no voters:
I20260810 15:00:27.533502 20374 leader_election.cc:288] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:00:27.533558 20379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:00:27.533669 20379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 LEADER]: Becoming Leader. State: Replica: 8c879231144442458856d07885aabc79, State: Running, Role: LEADER
I20260810 15:00:27.533809 20374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:27.533784 20379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:27.534145 20382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c879231144442458856d07885aabc79. Latest consensus state: current_term: 2 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:27.534292 20382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:27.534472 20380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:27.534576 20380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:27.534688 20385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:27.535462 20385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:27.535564 20385 catalog_manager.cc:1259] Loaded cluster ID: 6a9b1437305847c1a63abf18bc64659c
I20260810 15:00:27.535593 20385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:00:27.535818 20385 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 15:00:27.540891 20385 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 15:00:27.543864 20385 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:27.544077 20385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Loaded TSK: 0
I20260810 15:00:27.544286 20385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:00:27.609601 20377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:27.609850 20377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:27.609877 20377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:27.611068 20377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:27.611115 20377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:27.611140 20377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:27.612627 20377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:36127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=38019
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:27.612874 20377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:27.613102 20377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:27.618005 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.618002 20422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.618050 20425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.618391 20377 server_base.cc:1034] running on GCE node
I20260810 15:00:27.618587 20377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:27.618813 20377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:27.619977 20377 hybrid_clock.cc:648] HybridClock initialized: now 1786374027619965 us; error 27 us; skew 500 ppm
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:27.622716 20377 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:27.623191 20377 webserver.cc:460] Webserver started at http://127.28.193.193:38019/ using document root <none> and password file <none>
I20260810 15:00:27.623435 20377 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:27.623512 20377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:27.624822 20377 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:27.625438 20439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.625623 20377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:27.625747 20377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "9dcf8ffcce4c461baae4f234b01b657f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "e2940e8cf0684e6e37e7efe078bc7b36"
server_key_iv: "2b93b698e7b0b68c1cd6dd3daca1f3a1"
server_key_version: "encryptionkey@0"
I20260810 15:00:27.626183 20377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:27.642652 20377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:27.644997 20377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:27.645143 20377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:27.645392 20377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:27.645924 20458 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:00:27.647389 20377 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:00:27.647449 20377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:00:27.647480 20377 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:00:27.648290 20377 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:00:27.648329 20377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:00:27.648387 20458 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap starting.
I20260810 15:00:27.655082 20377 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:36127
I20260810 15:00:27.655263 20639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:36127 every 8 connection(s)
I20260810 15:00:27.655563 20377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:00:27.658430 20458 log.cc:824] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:27.658869 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 20377
I20260810 15:00:27.659184 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:44087
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=46021
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:27.661151 20458 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:00:27.661424 20458 tablet_bootstrap.cc:492] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap complete.
I20260810 15:00:27.662552 20458 ts_tablet_manager.cc:1397] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260810 15:00:27.663849 20458 raft_consensus.cc:348] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
I20260810 15:00:27.664145 20458 raft_consensus.cc:729] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcf8ffcce4c461baae4f234b01b657f, State: Initialized, Role: FOLLOWER
I20260810 15:00:27.664270 20458 consensus_queue.cc:260] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } } peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } }
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:27.665532 20458 ts_tablet_manager.cc:1428] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 15:00:27.669438 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.656400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:42051)
0810 15:00:27.658053 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.658105 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.658799 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.659474 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.659477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.659811 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:27.660065 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.660071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.661390 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.661391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.662242 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.662245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.663129 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.663130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.663270 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.663875 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:27.663886 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:27.665610 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:27.667792 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.667799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.667801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.668104 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.668391 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.668392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.668392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.668525 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.668975 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:27.668980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:27.669090 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.669172 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.669294 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":92,"threads_started":1}
I20260810 15:00:27.669591 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.656523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:42051 (local address 127.28.193.254:32821)
0810 15:00:27.656718 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.656728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.658248 (+ 1520us) server_negotiation.cc:407] Connection header received
0810 15:00:27.659064 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.659074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.659138 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.659257 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:27.660202 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.661281 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.662324 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.662900 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.663014 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.665756 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:27.665761 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:27.665761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:27.665774 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:27.667698 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.668196 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.668200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.668200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.668316 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.668596 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.668597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.668598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.668898 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:27.669002 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.669254 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.669397 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260810 15:00:27.670305 20640 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:27.670411 20640 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:27.670614 20640 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:27.671516 20313 ts_manager.cc:194] Registered new tserver with Master: 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127)
I20260810 15:00:27.672755 20313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:42051
I20260810 15:00:27.673738 20640 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 15:00:27.680548 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.672173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36127 (local address 127.0.0.1:41392)
0810 15:00:27.672333 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.672367 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.672431 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.673383 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.673389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.673707 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:27.673789 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.673791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.674700 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.674701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.675678 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.675681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.676086 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.676087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.676157 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.676565 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:27.676569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:27.677985 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:27.679531 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.679537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.679539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.679721 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.679908 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.679908 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.679909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.679966 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.680197 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:27.680204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:27.680254 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.680326 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.680450 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260810 15:00:27.680548 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.673067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.28.193.193:36127)
0810 15:00:27.673222 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.673226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.673236 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:00:27.673269 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.673271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.673285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.673361 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:27.673902 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.674629 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.675746 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.676026 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.676075 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.678062 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:27.678069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:27.678070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:27.678076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:27.679449 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.679792 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.679795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.679796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.679848 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.680016 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.680017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.680017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.680136 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:27.680204 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.680368 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.680450 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260810 15:00:27.681093 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ebecb089e2417886640dcf1b2dfc60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:27 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41392
I20260810 15:00:27.681668 20653 tablet_replica.cc:330] stopping tablet replica
I20260810 15:00:27.681806 20653 raft_consensus.cc:2229] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:27.681880 20653 raft_consensus.cc:2258] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:27.682211 20653 ts_tablet_manager.cc:1905] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:27.684520 20653 ts_tablet_manager.cc:1918] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:27.684599 20653 log.cc:1196] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/40ebecb089e2417886640dcf1b2dfc60
I20260810 15:00:27.684816 20653 ts_tablet_manager.cc:1939] T 40ebecb089e2417886640dcf1b2dfc60 P 9dcf8ffcce4c461baae4f234b01b657f: Deleting consensus metadata
I20260810 15:00:27.685312 20290 catalog_manager.cc:4841] TS 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127): tablet 40ebecb089e2417886640dcf1b2dfc60 (table test-table [id=f6afbc10904641998126216993cd019a]) successfully deleted
W20260810 15:00:27.748943 20649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:27.749143 20649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:27.749166 20649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:27.750311 20649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:27.750355 20649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:27.750387 20649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:27.751837 20649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:44087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=46021
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:27.752072 20649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:27.752302 20649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:27.757261 20661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.757302 20663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.757427 20649 server_base.cc:1034] running on GCE node
W20260810 15:00:27.757409 20660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.757702 20649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:27.758002 20649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:27.759150 20649 hybrid_clock.cc:648] HybridClock initialized: now 1786374027759137 us; error 32 us; skew 500 ppm
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:27.761616 20649 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:27.762174 20649 webserver.cc:460] Webserver started at http://127.28.193.194:46021/ using document root <none> and password file <none>
I20260810 15:00:27.762421 20649 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:27.762491 20649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:27.763901 20649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:00:27.764514 20670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.764670 20649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:27.764767 20649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d096bb99de4ba859bd54316d23605e14"
server_key_iv: "4edce5167aca84e3ddb6564da18cd4b4"
server_key_version: "encryptionkey@0"
I20260810 15:00:27.765110 20649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:27.794639 20649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:27.796492 20649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:27.796645 20649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:27.796895 20649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:27.797214 20649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:27.797256 20649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.797291 20649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:27.797317 20649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.803357 20649 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44087
I20260810 15:00:27.803408 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44087 every 8 connection(s)
I20260810 15:00:27.803763 20649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:00:27.807402 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 20649
I20260810 15:00:27.807688 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:39545
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=42123
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:27.812959 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.804466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:33807 (local address 127.28.193.254:32821)
0810 15:00:27.804530 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.804532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.804722 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:00:27.805209 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.805210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.805230 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.805290 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:27.806235 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.806732 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.807923 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.808142 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.808205 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.810529 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:27.810530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:27.810531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:27.810538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:27.811682 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.812050 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.812051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.812051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.812122 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.812335 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.812336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.812337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.812454 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:27.812511 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.812773 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.812869 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:00:27.813155 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.804442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:33807)
0810 15:00:27.804599 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.804641 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.805129 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.805345 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.805348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.805955 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:27.806144 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.806150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.806805 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.806806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.807833 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.807837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.808226 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.808227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.808367 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.808817 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:27.808827 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:27.810436 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:27.811753 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.811760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.811763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.811980 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.812178 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.812179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.812179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.812279 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.812509 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:27.812516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:27.812654 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.812746 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.812904 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260810 15:00:27.813601 20875 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:27.813732 20875 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:27.813939 20875 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:27.814370 20313 ts_manager.cc:194] Registered new tserver with Master: fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087)
I20260810 15:00:27.814851 20313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:33807
W20260810 15:00:27.890988 20880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:27.891183 20880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:27.891206 20880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:27.892334 20880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:27.892372 20880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:27.892396 20880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:27.893980 20880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:39545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=42123
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:27.894219 20880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:27.894433 20880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:27.899175 20890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:27.899220 20893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.899320 20880 server_base.cc:1034] running on GCE node
W20260810 15:00:27.899220 20891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:27.899571 20880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:27.899797 20880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:27.900944 20880 hybrid_clock.cc:648] HybridClock initialized: now 1786374027900926 us; error 28 us; skew 500 ppm
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:27.903412 20880 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:27.903831 20880 webserver.cc:460] Webserver started at http://127.28.193.195:42123/ using document root <none> and password file <none>
I20260810 15:00:27.904038 20880 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:27.904105 20880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:27.905370 20880 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:27.905939 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.906097 20880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:27.906181 20880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "2705ded81f5f4823a1bd7a7460ce1135"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d0fc02faf40e81e43db0e9fe27ec6fd4"
server_key_iv: "1aebd6b6a7d4091b1e9e93f1fa43c97d"
server_key_version: "encryptionkey@0"
I20260810 15:00:27.906507 20880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:27.932106 20880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:27.934563 20880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:27.934724 20880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:27.934963 20880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:27.935302 20880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:27.935338 20880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.935376 20880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:27.935397 20880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:27.940764 20880 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:39545
I20260810 15:00:27.940831 21127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:39545 every 8 connection(s)
I20260810 15:00:27.941148 20880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:00:27.944053 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 20880
Aug 10 15:00:27 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:27.950489 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.944430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.28.193.254:32821)
0810 15:00:27.944546 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.944547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.944556 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:00:27.944625 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.944626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.944636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.944690 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:27.945229 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.945865 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.946642 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.946856 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.946911 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.948061 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:27.948062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:27.948062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:27.948066 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:27.949394 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949749 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.949750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.949751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.949808 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949995 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.949995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.949995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.950101 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:27.950272 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.950309 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.950394 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260810 15:00:27.950598 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.941889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:49049 (local address 127.28.193.254:32821)
0810 15:00:27.941950 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.941952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.942152 (+ 200us) server_negotiation.cc:407] Connection header received
0810 15:00:27.942648 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.942649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.942661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.942718 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:27.943626 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.944131 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.945207 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.945420 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.945474 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.947667 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:27.947668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:27.947669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:27.947676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:27.949226 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949597 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.949598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.949598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.949655 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949924 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:27.949925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:27.949925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:27.950052 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:27.950104 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.950422 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.950495 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:00:27.950868 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.941886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:49049)
0810 15:00:27.942037 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.942073 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.942565 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.942775 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.942779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.943327 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:27.943531 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.943537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.944217 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.944218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.945120 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.945123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.945494 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.945495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.945645 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.946166 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:27.946175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:27.947583 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:27.949313 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.949321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.949534 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.949736 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:27.949736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:27.949737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:27.949867 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:27.950114 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:27.950120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:27.950272 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.950399 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.950551 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260810 15:00:27.951282 21128 heartbeater.cc:347] Connected to a master server at 127.28.193.254:32821
I20260810 15:00:27.951395 21128 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:27.951571 21128 heartbeater.cc:511] Master 127.28.193.254:32821 requested a full tablet report, sending...
I20260810 15:00:27.951917 20313 ts_manager.cc:194] Registered new tserver with Master: 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
I20260810 15:00:27.952227 20313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:49049
I20260810 15:00:27.962369 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:00:27.965039 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.962742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.28.193.254:32821)
0810 15:00:27.962809 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.962811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.962885 (+ 74us) server_negotiation.cc:407] Connection header received
0810 15:00:27.962969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.962970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.963009 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.963066 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:00:27.963153 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.963631 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.964393 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.964581 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.964636 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.964754 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:00:27.964852 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:00:27.964908 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.964937 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.964961 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:00:27.965325 20313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:32972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:00:27.966974 20313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:00:27.968947 20524 tablet_service.cc:1467] Processing CreateTablet for tablet dbb18aad3b754fe088fc4d770baed17a (DEFAULT_TABLE table=test-table [id=a94d99289b9143caaac05faf5e365dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:27.969161 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb18aad3b754fe088fc4d770baed17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:27.970507 20458 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap starting.
I20260810 15:00:27.970849 20458 tablet_bootstrap.cc:654] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:27.971428 20458 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: No bootstrap required, opened a new log
I20260810 15:00:27.971477 20458 ts_tablet_manager.cc:1397] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:00:27.971807 20458 raft_consensus.cc:348] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.971873 20458 raft_consensus.cc:374] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:27.971911 20458 raft_consensus.cc:729] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcf8ffcce4c461baae4f234b01b657f, State: Initialized, Role: FOLLOWER
I20260810 15:00:27.971966 20458 consensus_queue.cc:260] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.972043 20458 ts_tablet_manager.cc:1428] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 15:00:27.972244 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44087 (local address 127.0.0.1:57822)
0810 15:00:27.968742 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.968751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.968786 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.969193 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.969194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.969196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:27.969236 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.969238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.970228 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.970230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.971239 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.971242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.971986 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.971988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.972053 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.972072 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.972128 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.972177 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:00:27.972256 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.968807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.28.193.194:44087)
0810 15:00:27.969031 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.969035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.969046 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:27.969086 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.969088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.969102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.969174 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:27.969289 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.970148 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.971315 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.971910 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.972008 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.972073 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.972143 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.972178 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260810 15:00:27.973067 20753 tablet_service.cc:1467] Processing CreateTablet for tablet dbb18aad3b754fe088fc4d770baed17a (DEFAULT_TABLE table=test-table [id=a94d99289b9143caaac05faf5e365dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:27.973393 20753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb18aad3b754fe088fc4d770baed17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:27.974207 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.970559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.28.193.195:39545)
0810 15:00:27.970757 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:27.970762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:27.970773 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:27.970811 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:27.970813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:27.970827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:27.970923 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:27.971049 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.972062 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.973305 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.973885 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.973945 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.973981 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:27.974116 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 15:00:27.974145 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260810 15:00:27.974213 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:27.968719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39545 (local address 127.0.0.1:56140)
0810 15:00:27.968899 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:27.968907 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:27.968944 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:27.970936 (+ 1992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:27.970938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:27.970939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:27.970987 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.970989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.972183 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.972184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.973208 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:27.973211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:27.973962 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:27.973963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:27.974030 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:27.974034 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:27.974105 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:00:27.974147 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260810 15:00:27.974915 21001 tablet_service.cc:1467] Processing CreateTablet for tablet dbb18aad3b754fe088fc4d770baed17a (DEFAULT_TABLE table=test-table [id=a94d99289b9143caaac05faf5e365dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:27.975287 21001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb18aad3b754fe088fc4d770baed17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:27.976850 21143 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Bootstrap starting.
I20260810 15:00:27.977407 21143 tablet_bootstrap.cc:654] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:27.977962 21143 log.cc:824] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:27.978323 21146 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Bootstrap starting.
I20260810 15:00:27.978713 21143 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: No bootstrap required, opened a new log
I20260810 15:00:27.978768 21143 ts_tablet_manager.cc:1397] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:27.978904 21146 tablet_bootstrap.cc:654] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:27.979324 21146 log.cc:824] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:27.979948 21146 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: No bootstrap required, opened a new log
I20260810 15:00:27.980005 21146 ts_tablet_manager.cc:1397] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:00:27.980154 21143 raft_consensus.cc:348] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.980283 21143 raft_consensus.cc:374] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:27.980304 21143 raft_consensus.cc:729] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe85dd5d4de4477cafa3aca9e320d24f, State: Initialized, Role: FOLLOWER
I20260810 15:00:27.980414 21143 consensus_queue.cc:260] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.980681 21143 ts_tablet_manager.cc:1428] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:27.980708 20875 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
I20260810 15:00:27.981845 21146 raft_consensus.cc:348] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.981971 21146 raft_consensus.cc:374] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:27.981992 21146 raft_consensus.cc:729] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2705ded81f5f4823a1bd7a7460ce1135, State: Initialized, Role: FOLLOWER
I20260810 15:00:27.982071 21146 consensus_queue.cc:260] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:27.982264 21146 ts_tablet_manager.cc:1428] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:27.982306 21128 heartbeater.cc:503] Master 127.28.193.254:32821 was elected leader, sending a full tablet report...
W20260810 15:00:28.054319 20877 tablet.cc:2369] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:28.148059 21150 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:28.148193 21150 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:28.148620 21150 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545), 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127)
I20260810 15:00:28.151427 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.148879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:37267 (local address 127.28.193.193:36127)
0810 15:00:28.148937 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:28.148939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:28.148952 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:00:28.149056 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:28.149058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:28.149080 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:28.149137 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:28.149292 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.149791 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.150699 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.151114 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.151178 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:28.151260 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:28.151329 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:00:28.151368 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:00:28.151427 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.148804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36127 (local address 127.28.193.194:37267)
0810 15:00:28.148888 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.148903 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:28.148967 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:28.149151 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:28.149152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:28.149154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:28.149228 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:28.149230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.149865 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.149866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:28.150607 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:28.150609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.151189 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.151190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:28.151264 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:28.151268 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:28.151312 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:00:28.151351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 15:00:28.151546 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.148804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39545 (local address 127.28.193.194:56905)
0810 15:00:28.149104 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.149112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:28.149173 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:28.149312 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:28.149313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:28.149314 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:28.149386 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:28.149388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.150045 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.150045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:28.150765 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:28.150777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.151302 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.151303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:28.151373 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:28.151377 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:28.151424 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:00:28.151463 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":104,"thread_start_us":135,"threads_started":1}
I20260810 15:00:28.151538 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.148920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:56905 (local address 127.28.193.195:39545)
0810 15:00:28.149005 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:28.149007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:28.149178 (+ 171us) server_negotiation.cc:407] Connection header received
0810 15:00:28.149235 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:28.149236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:28.149245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:28.149307 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:28.149447 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.149961 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.150861 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.151230 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.151302 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:28.151309 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:28.151438 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 15:00:28.151475 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 15:00:28.151746 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb18aad3b754fe088fc4d770baed17a" candidate_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcf8ffcce4c461baae4f234b01b657f" is_pre_election: true
I20260810 15:00:28.151914 20562 raft_consensus.cc:2454] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe85dd5d4de4477cafa3aca9e320d24f in term 0.
I20260810 15:00:28.151969 21040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb18aad3b754fe088fc4d770baed17a" candidate_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705ded81f5f4823a1bd7a7460ce1135" is_pre_election: true
I20260810 15:00:28.152129 21040 raft_consensus.cc:2454] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe85dd5d4de4477cafa3aca9e320d24f in term 0.
I20260810 15:00:28.152158 20683 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f, fe85dd5d4de4477cafa3aca9e320d24f; no voters:
I20260810 15:00:28.152318 21150 raft_consensus.cc:2781] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:00:28.152369 21150 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:00:28.152390 21150 raft_consensus.cc:3037] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:28.153148 21150 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:28.153265 21150 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [CANDIDATE]: Term 1 election: Requested vote from peers 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545), 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127)
I20260810 15:00:28.153431 21040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb18aad3b754fe088fc4d770baed17a" candidate_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2705ded81f5f4823a1bd7a7460ce1135"
I20260810 15:00:28.153437 20562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb18aad3b754fe088fc4d770baed17a" candidate_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcf8ffcce4c461baae4f234b01b657f"
I20260810 15:00:28.153508 20562 raft_consensus.cc:3037] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:28.153512 21040 raft_consensus.cc:3037] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:28.154282 21040 raft_consensus.cc:2454] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe85dd5d4de4477cafa3aca9e320d24f in term 1.
I20260810 15:00:28.154285 20562 raft_consensus.cc:2454] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe85dd5d4de4477cafa3aca9e320d24f in term 1.
I20260810 15:00:28.154443 20683 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f, fe85dd5d4de4477cafa3aca9e320d24f; no voters:
I20260810 15:00:28.154565 21150 raft_consensus.cc:2781] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:28.154711 21150 raft_consensus.cc:686] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 1 LEADER]: Becoming Leader. State: Replica: fe85dd5d4de4477cafa3aca9e320d24f, State: Running, Role: LEADER
I20260810 15:00:28.154829 21150 consensus_queue.cc:237] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:28.155612 20312 catalog_manager.cc:5495] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f reported cstate change: term changed from 0 to 1, leader changed from <none> to fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194). New cstate: current_term: 1 leader_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } health_report { overall_health: UNKNOWN } } }
W20260810 15:00:28.158416 20641 tablet.cc:2369] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:28.191656 21129 tablet.cc:2369] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:28.202021 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.199775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.28.193.194:44087)
0810 15:00:28.199835 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:28.199837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:28.199845 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:00:28.199896 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:28.199897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:28.199916 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:28.199975 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:00:28.200088 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.200552 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.201249 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:28.201440 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:28.201494 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:28.201648 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:00:28.201794 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:00:28.201869 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:28.201904 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 15:00:28.201924 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:00:28.203878 20562 raft_consensus.cc:1261] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Refusing update from remote peer fe85dd5d4de4477cafa3aca9e320d24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:28.203874 21040 raft_consensus.cc:1261] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Refusing update from remote peer fe85dd5d4de4477cafa3aca9e320d24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:28.204237 21157 consensus_queue.cc:1035] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:28.204381 21150 consensus_queue.cc:1035] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:28.207443 21163 mvcc.cc:205] Tried to move back new op lower bound from 7316988019520139264 to 7316988019322626048. Current Snapshot: MvccSnapshot[applied={T|T < 7316988019520139264 or (T in {7316988019520139264})}]
I20260810 15:00:28.209239 20313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:28.209327 20313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:32972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:28.210019 20313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Sending DeleteTablet for 3 replicas of tablet dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:28.210325 21001 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb18aad3b754fe088fc4d770baed17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:28 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:56140
I20260810 15:00:28.210347 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb18aad3b754fe088fc4d770baed17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:28 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41392
I20260810 15:00:28.210350 20753 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb18aad3b754fe088fc4d770baed17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:28 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:57822
I20260810 15:00:28.210551 21177 tablet_replica.cc:330] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: stopping tablet replica
I20260810 15:00:28.210611 21177 raft_consensus.cc:2229] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:28.210615 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 20377
I20260810 15:00:28.210665 21177 raft_consensus.cc:2258] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:28.210779 21179 tablet_replica.cc:330] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: stopping tablet replica
I20260810 15:00:28.210860 21179 raft_consensus.cc:2229] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:28.210935 21179 raft_consensus.cc:2258] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:28.213899 21179 ts_tablet_manager.cc:1905] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:28.215354 21179 ts_tablet_manager.cc:1918] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:28.215410 21179 log.cc:1196] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:28.215653 21179 ts_tablet_manager.cc:1939] T dbb18aad3b754fe088fc4d770baed17a P 2705ded81f5f4823a1bd7a7460ce1135: Deleting consensus metadata
I20260810 15:00:28.216809 20288 catalog_manager.cc:4841] TS 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): tablet dbb18aad3b754fe088fc4d770baed17a (table test-table [id=a94d99289b9143caaac05faf5e365dc8]) successfully deleted
I20260810 15:00:28.210686 21175 tablet_replica.cc:330] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: stopping tablet replica
I20260810 15:00:28.216990 21175 raft_consensus.cc:2229] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:00:28.217115 21175 raft_consensus.cc:2258] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:28.219081 21175 ts_tablet_manager.cc:1905] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:28.220494 21175 ts_tablet_manager.cc:1918] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:28.220557 21175 log.cc:1196] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:28.220781 21175 ts_tablet_manager.cc:1939] T dbb18aad3b754fe088fc4d770baed17a P fe85dd5d4de4477cafa3aca9e320d24f: Deleting consensus metadata
I20260810 15:00:28.221202 20286 catalog_manager.cc:4841] TS fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): tablet dbb18aad3b754fe088fc4d770baed17a (table test-table [id=a94d99289b9143caaac05faf5e365dc8]) successfully deleted
W20260810 15:00:28.235292 20290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:36127 (error 108)
I20260810 15:00:28.235733 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 20649
I20260810 15:00:28.235855 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.235743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36127 (local address 127.0.0.1:41398)
0810 15:00:28.235780 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.235821 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 15:00:28.237584 20290 catalog_manager.cc:4584] TS 9dcf8ffcce4c461baae4f234b01b657f (127.28.193.193:36127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbb18aad3b754fe088fc4d770baed17a: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36127: connect: Connection refused (error 111)
I20260810 15:00:28.243307 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 20880
I20260810 15:00:28.255188 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 20251
I20260810 15:00:28.265303 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--webserver_interface=127.28.193.254
--webserver_port=34159
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:28.347079 21183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:28.347285 21183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:28.347313 21183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:28.348448 21183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:28.348490 21183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:28.348510 21183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:28.348524 21183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:28.348539 21183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:28.350088 21183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=34159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:28.350337 21183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:28.350540 21183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:28.355239 21195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.355294 21198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.355319 21200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.355356 21183 server_base.cc:1034] running on GCE node
I20260810 15:00:28.355736 21183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:28.356036 21183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:28.357191 21183 hybrid_clock.cc:648] HybridClock initialized: now 1786374028357174 us; error 29 us; skew 500 ppm
Aug 10 15:00:28 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:28.359807 21183 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:28.360247 21183 webserver.cc:460] Webserver started at http://127.28.193.254:34159/ using document root <none> and password file <none>
I20260810 15:00:28.360455 21183 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:28.360502 21183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:28.361825 21183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:00:28.362458 21210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.362605 21183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:28.362687 21183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "8c879231144442458856d07885aabc79"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "b62e0f4e4013eb694a1d6ee3831a9adb"
server_key_iv: "6602c9bd4410f35de55ef8a6db53a8df"
server_key_version: "encryptionkey@0"
I20260810 15:00:28.363056 21183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:28.391885 21183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:28.394141 21183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:28.394290 21183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:28.398437 21183 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:32821
I20260810 15:00:28.398521 21303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:32821 every 8 connection(s)
I20260810 15:00:28.398893 21183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:28.400452 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 21183
I20260810 15:00:28.400604 21305 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:00:28.400779 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:36127
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=38019
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:28.401477 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap starting.
I20260810 15:00:28.409757 21305 log.cc:824] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:28.412106 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:00:28.412388 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79: Bootstrap complete.
I20260810 15:00:28.414593 21305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:28.414824 21305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c879231144442458856d07885aabc79, State: Initialized, Role: FOLLOWER
I20260810 15:00:28.414985 21305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:28.415052 21305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:28.415092 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:28.415139 21305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:00:28.416126 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:28.416225 21305 leader_election.cc:302] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c879231144442458856d07885aabc79; no voters:
I20260810 15:00:28.416388 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:00:28.416436 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:00:28.416548 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [term 3 LEADER]: Becoming Leader. State: Replica: 8c879231144442458856d07885aabc79, State: Running, Role: LEADER
I20260810 15:00:28.416661 21305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:28.416640 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } }
I20260810 15:00:28.416944 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:28.417033 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:28.417253 21319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c879231144442458856d07885aabc79. Latest consensus state: current_term: 3 leader_uuid: "8c879231144442458856d07885aabc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c879231144442458856d07885aabc79" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 32821 } } }
I20260810 15:00:28.417354 21319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c879231144442458856d07885aabc79 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:28.417399 21323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:28.418257 21323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:28.418368 21323 catalog_manager.cc:1259] Loaded cluster ID: 6a9b1437305847c1a63abf18bc64659c
I20260810 15:00:28.418402 21323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 15:00:28.418764 21323 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08c1a499ad google::LogMessage::Fail() at ??:0
@ 0x7f08c1a4c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f08c1a494de google::LogMessage::Flush() at ??:0
@ 0x7f08c1a4aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f08c996e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f08c99778b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f08c39170d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f08c390c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f08c47706db start_thread at ??:0
@ 0x7f08c054761f clone at ??:0
W20260810 15:00:28.497478 21309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:28.497671 21309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:28.497694 21309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:28.498834 21309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:28.498874 21309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:28.498898 21309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:28.500351 21309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:36127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=38019
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:28.500581 21309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:28.500792 21309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:28.506691 21368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.506691 21363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.506723 21362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.506999 21309 server_base.cc:1034] running on GCE node
I20260810 15:00:28.507231 21309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:28.507510 21309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:28.508694 21309 hybrid_clock.cc:648] HybridClock initialized: now 1786374028508669 us; error 35 us; skew 500 ppm
Aug 10 15:00:28 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:28.511801 21309 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:28.512372 21309 webserver.cc:460] Webserver started at http://127.28.193.193:38019/ using document root <none> and password file <none>
I20260810 15:00:28.512645 21309 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:28.512718 21309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:28.514515 21309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:00:28.515268 21378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.515446 21309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:28.515535 21309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "9dcf8ffcce4c461baae4f234b01b657f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "e2940e8cf0684e6e37e7efe078bc7b36"
server_key_iv: "2b93b698e7b0b68c1cd6dd3daca1f3a1"
server_key_version: "encryptionkey@0"
I20260810 15:00:28.515864 21309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:28.544237 21309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:28.546638 21309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:28.546788 21309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:28.547055 21309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:28.547542 21387 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:00:28.549445 21309 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:00:28.549506 21309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:00:28.549544 21309 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:00:28.550283 21309 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:00:28.550320 21309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:00:28.550407 21387 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap starting.
I20260810 15:00:28.558027 21309 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:36127
I20260810 15:00:28.558550 21309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:00:28.559580 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 21309
I20260810 15:00:28.559809 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:44087
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=46021
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:00:28.561781 21588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:36127 every 8 connection(s)
I20260810 15:00:28.561568 21387 log.cc:824] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:28.571667 21387 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:00:28.571987 21387 tablet_bootstrap.cc:492] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Bootstrap complete.
I20260810 15:00:28.573113 21387 ts_tablet_manager.cc:1397] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.011s
I20260810 15:00:28.574599 21387 raft_consensus.cc:348] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:28.574944 21387 raft_consensus.cc:729] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcf8ffcce4c461baae4f234b01b657f, State: Initialized, Role: FOLLOWER
I20260810 15:00:28.575115 21387 consensus_queue.cc:260] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:28.575415 21387 ts_tablet_manager.cc:1428] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
W20260810 15:00:28.610284 21593 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:00:28.562365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:35839)
0810 15:00:28.565211 (+ 2846us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.565284 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:28.565957 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:28.610057 (+ 44100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2785,"thread_start_us":116,"threads_started":1}
W20260810 15:00:28.610558 21589 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:32821 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:32821: Client connection negotiation failed: client connection to 127.28.193.254:32821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:00:28.610893 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.610772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:50823)
0810 15:00:28.610825 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.610868 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:00:28.611209 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.611127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:55347)
0810 15:00:28.611166 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.611191 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:00:28.611366 21589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:00:28.649581 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:28.649829 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:28.649859 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:28.651007 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:28.651048 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:28.651072 21592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:28.652498 21592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:44087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=46021
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:28.652715 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:28.652940 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:28.657843 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.657838 21608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.658155 21592 server_base.cc:1034] running on GCE node
W20260810 15:00:28.658151 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.658409 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:28.658654 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:28.659788 21592 hybrid_clock.cc:648] HybridClock initialized: now 1786374028659775 us; error 31 us; skew 500 ppm
Aug 10 15:00:28 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:28.662310 21592 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:28.662760 21592 webserver.cc:460] Webserver started at http://127.28.193.194:46021/ using document root <none> and password file <none>
I20260810 15:00:28.663023 21592 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:28.663074 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:28.664330 21592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:28.664856 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.665012 21592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.665094 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "fe85dd5d4de4477cafa3aca9e320d24f"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d096bb99de4ba859bd54316d23605e14"
server_key_iv: "4edce5167aca84e3ddb6564da18cd4b4"
server_key_version: "encryptionkey@0"
I20260810 15:00:28.665436 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:28.681187 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:28.683425 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:28.683550 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:28.683751 21592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:28.684062 21592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:28.684100 21592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.684122 21592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:28.684145 21592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.689770 21592 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:44087
I20260810 15:00:28.689833 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:44087 every 8 connection(s)
I20260810 15:00:28.690157 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:00:28.691272 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.690828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:35267)
0810 15:00:28.691024 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.691124 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
W20260810 15:00:28.691691 21837 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:32821 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:32821: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
I20260810 15:00:28.691941 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.691854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:59023)
0810 15:00:28.691896 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.691923 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 15:00:28.692148 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.692088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:38651)
0810 15:00:28.692118 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.692136 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 15:00:28.692229 21837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:00:28.697798 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 21592
I20260810 15:00:28.698069 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:39545
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:32821
--webserver_port=42123
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:28.779819 21843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:28.780028 21843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:28.780052 21843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:28.781188 21843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:28.781226 21843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:28.781251 21843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:28.782737 21843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:39545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=42123
--tserver_master_addrs=127.28.193.254:32821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:28.782985 21843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:28.783200 21843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:28.788105 21853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:28.788151 21859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.788188 21843 server_base.cc:1034] running on GCE node
W20260810 15:00:28.788115 21852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:28.788514 21843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:28.788727 21843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:28.789860 21843 hybrid_clock.cc:648] HybridClock initialized: now 1786374028789833 us; error 44 us; skew 500 ppm
Aug 10 15:00:28 dist-test-slave-cvwc krb5kdc[19294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:28.792250 21843 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:28.792672 21843 webserver.cc:460] Webserver started at http://127.28.193.195:42123/ using document root <none> and password file <none>
I20260810 15:00:28.792878 21843 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:28.792918 21843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:28.794193 21843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:00:28.794697 21870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.794857 21843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.794934 21843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "2705ded81f5f4823a1bd7a7460ce1135"
format_stamp: "Formatted at 2026-08-10 15:00:26 on dist-test-slave-cvwc"
server_key: "d0fc02faf40e81e43db0e9fe27ec6fd4"
server_key_iv: "1aebd6b6a7d4091b1e9e93f1fa43c97d"
server_key_version: "encryptionkey@0"
I20260810 15:00:28.795265 21843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:28.808091 21843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:28.810073 21843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:28.810197 21843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:28.810393 21843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:28.810678 21843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:28.810708 21843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.810729 21843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:28.810750 21843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:28.816193 21843 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:39545
I20260810 15:00:28.816236 22079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:39545 every 8 connection(s)
I20260810 15:00:28.816545 21843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:00:28.817483 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.817138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:58697)
0810 15:00:28.817268 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.817339 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260810 15:00:28.817880 22081 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:32821 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:32821: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
I20260810 15:00:28.818112 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.818033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:49649)
0810 15:00:28.818070 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.818095 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:00:28.818310 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:28.818252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:45287)
0810 15:00:28.818281 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:28.818298 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 15:00:28.818388 22081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:00:28.823424 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 21843
I20260810 15:00:29.611976 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:29.611726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:58155)
0810 15:00:29.611861 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:29.611901 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260810 15:00:29.692719 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:29.692521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:44853)
0810 15:00:29.692635 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:29.692669 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260810 15:00:29.818872 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:29.818687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:43941)
0810 15:00:29.818794 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:29.818826 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
I20260810 15:00:30.338158 22100 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:30.338284 22100 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:30.338646 22100 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
Aug 10 15:00:30 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.195@KRBTEST.COM
Aug 10 15:00:30 dist-test-slave-cvwc krb5kdc[19294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.194@KRBTEST.COM
I20260810 15:00:30.349488 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.338790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:39545 (local address 127.28.193.193:52327)
0810 15:00:30.339064 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:30.339074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:30.339140 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:30.339795 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:30.339797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:30.340171 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:30.340347 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:30.340354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.341506 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.341507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:30.342675 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:30.342678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.343413 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.343414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:30.343546 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:30.344076 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:30.344086 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:30.345814 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:30.348014 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:30.348024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:30.348304 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:30.348540 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:30.348542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:30.348644 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:30.348993 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:30.349000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:30.349123 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 15:00:30.349205 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:00:30.349373 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":33,"thread_start_us":58,"threads_started":1}
I20260810 15:00:30.349524 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.338922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:42095 (local address 127.28.193.194:44087)
0810 15:00:30.339075 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:30.339084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:30.339102 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:00:30.339563 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:30.339565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:30.339607 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:30.339738 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:30.340467 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.341399 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.342910 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.343498 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.343598 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:30.346253 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:30.346257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:30.346257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:30.346268 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:30.348023 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348374 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:30.348377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:30.348377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:30.348468 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348703 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:30.348703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:30.348704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:30.348960 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:30.349041 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:30.349280 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 15:00:30.349416 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":45,"threads_started":1}
I20260810 15:00:30.349565 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.338789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:44087 (local address 127.28.193.193:42095)
0810 15:00:30.338910 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:30.338921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:30.338981 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:30.339795 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:30.339797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:30.340169 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:30.340347 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:30.340353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.341491 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.341493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:30.342671 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:30.342674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.343576 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.343577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:30.343639 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:30.344216 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:30.344221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:30.346159 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:30.348099 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:30.348100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:30.348305 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:30.348529 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:30.348530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:30.348643 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:30.349035 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:30.349036 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:30.349119 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 15:00:30.349204 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:00:30.349444 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":186,"thread_start_us":53,"threads_started":1}
I20260810 15:00:30.350049 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.338930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:52327 (local address 127.28.193.195:39545)
0810 15:00:30.339064 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:30.339078 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:30.339125 (+ 47us) server_negotiation.cc:407] Connection header received
0810 15:00:30.339595 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:30.339598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:30.339635 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:30.339755 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:30.340469 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.341369 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.342752 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:30.343329 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:30.343447 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:30.345927 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:30.345932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:30.345932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:30.345944 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:30.347937 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348389 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:30.348392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:30.348393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:30.348477 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:30.348705 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:30.348706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:30.348706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:30.348931 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:30.348991 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:30.349798 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 15:00:30.349946 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
W20260810 15:00:30.350312 21379 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
W20260810 15:00:30.350697 21381 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:30.350754 21381 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f; no voters: 2705ded81f5f4823a1bd7a7460ce1135, fe85dd5d4de4477cafa3aca9e320d24f
I20260810 15:00:30.350853 22100 raft_consensus.cc:2726] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:00:30.612558 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.612425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:38573)
0810 15:00:30.612471 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:30.612515 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:00:30.693324 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.693091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:59685)
0810 15:00:30.693233 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:30.693278 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260810 15:00:30.819506 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:30.819271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:45541)
0810 15:00:30.819404 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:30.819456 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260810 15:00:31.613183 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:31.612962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:54281)
0810 15:00:31.613098 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:31.613135 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260810 15:00:31.693964 22121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:31.693757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:40515)
0810 15:00:31.693875 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:31.693909 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260810 15:00:31.820122 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:31.819919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:47207)
0810 15:00:31.820040 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:31.820078 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260810 15:00:32.469415 22127 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:32.469513 22127 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:32.469661 22127 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
W20260810 15:00:32.470184 21379 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
W20260810 15:00:32.470279 21381 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:32.470359 21381 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f; no voters: 2705ded81f5f4823a1bd7a7460ce1135, fe85dd5d4de4477cafa3aca9e320d24f
I20260810 15:00:32.470438 22127 raft_consensus.cc:2726] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:00:32.613881 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:32.613634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:44813)
0810 15:00:32.613789 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:32.613825 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260810 15:00:32.694674 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:32.694417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:46345)
0810 15:00:32.694553 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:32.694597 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260810 15:00:32.820818 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:32.820591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:47929)
0810 15:00:32.820718 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:32.820757 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260810 15:00:33.614702 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:33.614424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:41821)
0810 15:00:33.614598 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:33.614651 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260810 15:00:33.695292 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:33.695085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:43919)
0810 15:00:33.695215 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:33.695251 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260810 15:00:33.821503 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:33.821281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:57683)
0810 15:00:33.821420 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:33.821460 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260810 15:00:34.300684 22142 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:34.300839 22142 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:34.300997 22142 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
W20260810 15:00:34.301385 21381 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
W20260810 15:00:34.301478 21379 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:34.301541 21379 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f; no voters: 2705ded81f5f4823a1bd7a7460ce1135, fe85dd5d4de4477cafa3aca9e320d24f
I20260810 15:00:34.301608 22142 raft_consensus.cc:2726] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:00:34.615379 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:34.615165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:57839)
0810 15:00:34.615297 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:34.615330 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260810 15:00:34.695979 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:34.695769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:38605)
0810 15:00:34.695893 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:34.695933 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260810 15:00:34.822137 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:34.821926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:51721)
0810 15:00:34.822056 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:34.822094 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260810 15:00:35.616236 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:35.615931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:54147)
0810 15:00:35.616144 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:35.616189 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":148,"threads_started":1}
I20260810 15:00:35.696694 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:35.696455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:48407)
0810 15:00:35.696604 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:35.696648 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260810 15:00:35.822690 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:35.822503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:56121)
0810 15:00:35.822611 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:35.822646 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260810 15:00:36.616912 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:36.616681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:41589)
0810 15:00:36.616822 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:36.616863 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260810 15:00:36.697328 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:36.697129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:43783)
0810 15:00:36.697255 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:36.697285 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260810 15:00:36.823249 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:36.823065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:45065)
0810 15:00:36.823178 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:36.823207 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260810 15:00:37.617615 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:37.617378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:44453)
0810 15:00:37.617530 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:37.617566 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260810 15:00:37.698040 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:37.697822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:40083)
0810 15:00:37.697958 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:37.697994 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260810 15:00:37.823840 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:37.823631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:38247)
0810 15:00:37.823762 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:37.823800 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260810 15:00:38.369827 22173 raft_consensus.cc:482] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:38.369931 22173 raft_consensus.cc:504] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe85dd5d4de4477cafa3aca9e320d24f" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 44087 } } peers { permanent_uuid: "2705ded81f5f4823a1bd7a7460ce1135" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 39545 } } peers { permanent_uuid: "9dcf8ffcce4c461baae4f234b01b657f" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36127 } }
I20260810 15:00:38.370100 22173 leader_election.cc:288] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087), 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545)
W20260810 15:00:38.370473 21381 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2705ded81f5f4823a1bd7a7460ce1135 (127.28.193.195:39545): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
W20260810 15:00:38.370553 21379 leader_election.cc:341] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe85dd5d4de4477cafa3aca9e320d24f (127.28.193.194:44087): Not found: Tablet not found: dbb18aad3b754fe088fc4d770baed17a
I20260810 15:00:38.370611 21379 leader_election.cc:302] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9dcf8ffcce4c461baae4f234b01b657f; no voters: 2705ded81f5f4823a1bd7a7460ce1135, fe85dd5d4de4477cafa3aca9e320d24f
I20260810 15:00:38.370693 22173 raft_consensus.cc:2726] T dbb18aad3b754fe088fc4d770baed17a P 9dcf8ffcce4c461baae4f234b01b657f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:00:38.618275 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:38.618043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:35321)
0810 15:00:38.618188 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:38.618228 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260810 15:00:38.698688 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:38.698440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:53065)
0810 15:00:38.698583 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:38.698622 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260810 15:00:38.824481 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:38.824247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:32817)
0810 15:00:38.824395 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:38.824434 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260810 15:00:39.618983 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:39.618760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:38291)
0810 15:00:39.618890 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:39.618932 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260810 15:00:39.699317 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:39.699132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:50115)
0810 15:00:39.699243 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:39.699273 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260810 15:00:39.825089 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:39.824888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:58633)
0810 15:00:39.825015 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:39.825048 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260810 15:00:40.619597 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:40.619395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:52449)
0810 15:00:40.619515 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:40.619548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260810 15:00:40.699937 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:40.699702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:47227)
0810 15:00:40.699834 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:40.699876 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260810 15:00:40.825706 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:40.825484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:39899)
0810 15:00:40.825621 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:40.825659 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260810 15:00:41.620290 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:41.620057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:40739)
0810 15:00:41.620207 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:41.620238 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260810 15:00:41.700585 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:41.700356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:37611)
0810 15:00:41.700498 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:41.700538 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260810 15:00:41.826324 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:41.826133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:53607)
0810 15:00:41.826247 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:41.826281 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260810 15:00:42.621001 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:42.620783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:42795)
0810 15:00:42.620914 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:42.620950 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260810 15:00:42.701197 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:42.700987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:51993)
0810 15:00:42.701115 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:42.701151 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260810 15:00:42.826965 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:42.826745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:48161)
0810 15:00:42.826880 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:42.826919 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260810 15:00:43.621754 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:43.621537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.193:51319)
0810 15:00:43.621650 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:43.621683 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260810 15:00:43.701925 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:43.701678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.194:40321)
0810 15:00:43.701833 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:43.701870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260810 15:00:43.827698 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:43.827478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:32821 (local address 127.28.193.195:47969)
0810 15:00:43.827617 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:43.827653 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:32821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260810 15:00:43.832648 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 21309
I20260810 15:00:43.844404 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 21592
I20260810 15:00:43.859475 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 21843
2026-08-10T15:00:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18978 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:00:43 dist-test-slave-cvwc krb5kdc[22211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:00:43 dist-test-slave-cvwc krb5kdc[22211](info): set up 2 sockets
Aug 10 15:00:43 dist-test-slave-cvwc krb5kdc[22211](info): commencing operation
krb5kdc: starting...
W20260810 15:00:45.035499 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:00:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:00:45.156388 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:45.238229 22237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:45.238454 22237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:45.238500 22237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:45.239660 22237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:00:45.239699 22237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:00:45.239714 22237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:00:45.239728 22237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:00:45.239744 22237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:45.239763 22237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:45.239781 22237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:00:45.239804 22237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:00:45.241304 22237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:45.241560 22237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:45.241792 22237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:00:45.246618 22237 server_base.cc:1034] running on GCE node
W20260810 15:00:45.246562 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.246560 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.246569 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.247010 22237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:45.247308 22237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:45.248449 22237 hybrid_clock.cc:648] HybridClock initialized: now 1786374045248437 us; error 28 us; skew 500 ppm
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:45.254997 22237 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:00:45.255473 22237 webserver.cc:460] Webserver started at http://127.28.193.254:45975/ using document root <none> and password file <none>
I20260810 15:00:45.255683 22237 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:45.255730 22237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:45.255860 22237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:45.256996 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "d0318d79c73d4cfc874cf3e254fd8c70"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "941cf05b71b7ca9b43c15be11aae3d02"
server_key_iv: "9b808bc66c5e2754cf1da294aebe19b8"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.257349 22237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "d0318d79c73d4cfc874cf3e254fd8c70"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "941cf05b71b7ca9b43c15be11aae3d02"
server_key_iv: "9b808bc66c5e2754cf1da294aebe19b8"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.258735 22237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:00:45.259445 22268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.259598 22237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:45.259686 22237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "d0318d79c73d4cfc874cf3e254fd8c70"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "941cf05b71b7ca9b43c15be11aae3d02"
server_key_iv: "9b808bc66c5e2754cf1da294aebe19b8"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.259747 22237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:45.284368 22237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:45.286248 22237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:45.286391 22237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:45.290333 22237 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:44961
I20260810 15:00:45.290378 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:44961 every 8 connection(s)
I20260810 15:00:45.290719 22237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:00:45.291612 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:45.291680 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 22237
I20260810 15:00:45.291782 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:00:45.291990 29447 external_mini_cluster.cc:1428] Setting key be36da715b9de0b169eb71cb30841728
I20260810 15:00:45.295009 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap starting.
I20260810 15:00:45.295719 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:45.296041 22357 log.cc:824] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:45.296756 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: No bootstrap required, opened a new log
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:45.299057 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:00:45.299204 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:45.299232 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Initialized, Role: FOLLOWER
I20260810 15:00:45.299361 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:00:45.299428 22357 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:00:45.299453 22357 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:00:45.299504 22357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:45.300096 22357 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:00:45.300194 22357 leader_election.cc:302] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0318d79c73d4cfc874cf3e254fd8c70; no voters:
I20260810 15:00:45.300348 22357 leader_election.cc:288] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:00:45.300380 22367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:45.300499 22367 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 LEADER]: Becoming Leader. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Running, Role: LEADER
I20260810 15:00:45.300637 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:00:45.300668 22367 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:00:45.301000 22367 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0318d79c73d4cfc874cf3e254fd8c70. Latest consensus state: current_term: 1 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:00:45.301077 22367 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:45.301290 22372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:00:45.301290 22368 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:00:45.301682 22368 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:00:45.301919 22372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:00:45.302585 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.292527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.28.193.254:44961)
0810 15:00:45.292688 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.292698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.292725 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:00:45.293231 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.293239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.293291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.293415 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:45.294197 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.295061 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.296073 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.296498 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.296579 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.299067 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:45.299071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:45.299072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:45.299082 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:45.300903 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.301257 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.301260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.301261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.301375 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.301594 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.301594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.301595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.301947 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:45.302127 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.302217 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.302368 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260810 15:00:45.303836 22372 catalog_manager.cc:1347] Generated new cluster ID: 45c33dc6c84148de89e87b77ed4d3c6f
I20260810 15:00:45.303874 22372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:00:45.313148 22372 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:00:45.313653 22372 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:00:45.327710 22372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Generated new TSK 0
I20260810 15:00:45.327893 22372 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:00:45.360754 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:45.444299 22400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:45.444501 22400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:45.444522 22400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:45.445662 22400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:45.445701 22400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:45.445753 22400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:00:45.447223 22400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:45.447428 22400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:45.447613 22400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:45.452488 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.452477 22409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.452477 22412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.452692 22400 server_base.cc:1034] running on GCE node
I20260810 15:00:45.452890 22400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:45.453130 22400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:45.454288 22400 hybrid_clock.cc:648] HybridClock initialized: now 1786374045454271 us; error 34 us; skew 500 ppm
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:45.456943 22400 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:00:45.457394 22400 webserver.cc:460] Webserver started at http://127.28.193.193:42339/ using document root <none> and password file <none>
I20260810 15:00:45.457620 22400 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:45.457669 22400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:45.457839 22400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:45.458891 22400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "380b5ee88abd43808c61196b03fb8b8b"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "139da8dfad80d80a3af4e5bbc973ae26"
server_key_iv: "b8ffd31dbb3c58f03d0470d2001a0f5e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.459229 22400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "380b5ee88abd43808c61196b03fb8b8b"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "139da8dfad80d80a3af4e5bbc973ae26"
server_key_iv: "b8ffd31dbb3c58f03d0470d2001a0f5e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.460496 22400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:00:45.461200 22427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.461366 22400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:00:45.461454 22400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "380b5ee88abd43808c61196b03fb8b8b"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "139da8dfad80d80a3af4e5bbc973ae26"
server_key_iv: "b8ffd31dbb3c58f03d0470d2001a0f5e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.461516 22400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:45.481374 22400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:45.483614 22400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:45.483765 22400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:45.484004 22400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:45.484329 22400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:45.484367 22400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.484397 22400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:45.484414 22400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.490082 22400 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:43729
I20260810 15:00:45.490137 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:43729 every 8 connection(s)
I20260810 15:00:45.490455 22400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:45.496622 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 22400
I20260810 15:00:45.496726 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:00:45.496956 29447 external_mini_cluster.cc:1428] Setting key 39b782f587aaf22010decf91e359840c
I20260810 15:00:45.499905 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.491189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:60655 (local address 127.28.193.254:44961)
0810 15:00:45.491235 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.491237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.491430 (+ 193us) server_negotiation.cc:407] Connection header received
0810 15:00:45.491947 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.491949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.491965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.492031 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:45.492983 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.493563 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.494658 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.494874 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.494933 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.497196 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:45.497198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:45.497198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:45.497206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:45.498542 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.498969 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.498970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.498970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.499016 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.499281 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.499282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.499283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.499420 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:45.499488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.499716 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.499806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:00:45.499976 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.491136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:60655)
0810 15:00:45.491303 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.491347 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.491834 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.492097 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.492100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.492661 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:45.492877 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.492883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.493666 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.493667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.494568 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.494571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.494967 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.494968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.495086 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.495536 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:45.495544 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:45.497077 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:45.498620 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.498627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.498630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.498905 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.499065 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.499066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.499066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.499202 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.499483 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:45.499487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:45.499602 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.499683 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.499804 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260810 15:00:45.500428 22625 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:00:45.500522 22625 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:45.500701 22625 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:00:45.501307 22300 ts_manager.cc:194] Registered new tserver with Master: 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729)
I20260810 15:00:45.502345 22300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:60655
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:00:45.522096 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:45.604391 22631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:45.604607 22631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:45.604631 22631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:45.605764 22631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:45.605800 22631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:45.605825 22631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:00:45.607235 22631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:45.607432 22631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:45.607615 22631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:45.612578 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.612591 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.612676 22631 server_base.cc:1034] running on GCE node
W20260810 15:00:45.612816 22638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.613049 22631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:45.613296 22631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:45.614465 22631 hybrid_clock.cc:648] HybridClock initialized: now 1786374045614450 us; error 30 us; skew 500 ppm
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:45.617236 22631 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:00:45.617748 22631 webserver.cc:460] Webserver started at http://127.28.193.194:44439/ using document root <none> and password file <none>
I20260810 15:00:45.617972 22631 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:45.618023 22631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:45.618147 22631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:45.619259 22631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "b6cbabf710240d185d1b2a4f59853b16"
server_key_iv: "8ae54c31d525a0933487c57958b4d34e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.619623 22631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "b6cbabf710240d185d1b2a4f59853b16"
server_key_iv: "8ae54c31d525a0933487c57958b4d34e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.620961 22631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:00:45.621658 22659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.621825 22631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:45.621912 22631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "b6cbabf710240d185d1b2a4f59853b16"
server_key_iv: "8ae54c31d525a0933487c57958b4d34e"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.621979 22631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:45.631814 22631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:45.633800 22631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:45.633949 22631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:45.634187 22631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:45.634507 22631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:45.634542 22631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.634572 22631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:45.634596 22631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.640278 22631 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:46119
I20260810 15:00:45.640326 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:46119 every 8 connection(s)
I20260810 15:00:45.640647 22631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:45.647342 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 22631
I20260810 15:00:45.647454 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:00:45.647635 29447 external_mini_cluster.cc:1428] Setting key 9ce181dd3a0e27327731006573af113c
I20260810 15:00:45.649325 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.641380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:53241 (local address 127.28.193.254:44961)
0810 15:00:45.641454 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.641455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.641621 (+ 166us) server_negotiation.cc:407] Connection header received
0810 15:00:45.642126 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.642128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.642138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.642196 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:45.643093 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.643582 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.644520 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.644698 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.644742 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.646713 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:45.646714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:45.646715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:45.646721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:45.648006 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.648404 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.648405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.648406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.648458 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.648704 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.648705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.648705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.648797 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:45.648851 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.649197 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.649263 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:00:45.649557 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.641342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:53241)
0810 15:00:45.641496 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.641538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.642042 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.642252 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.642256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.642814 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:45.643014 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.643019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.643656 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.643656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.644456 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.644458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.644753 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.644753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.644851 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.645283 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:45.645291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:45.646628 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:45.648077 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.648083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.648085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.648332 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.648512 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.648512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.648513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.648648 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.648844 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:45.648849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:45.649031 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.649174 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.649331 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260810 15:00:45.650274 22862 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:00:45.650357 22862 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:45.650523 22862 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:00:45.650887 22300 ts_manager.cc:194] Registered new tserver with Master: 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119)
I20260810 15:00:45.651393 22300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:53241
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:00:45.673220 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:00:45.757095 22871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:00:45.757304 22871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:00:45.757337 22871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:00:45.758590 22871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:00:45.758635 22871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:00:45.758670 22871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:00:45.760125 22871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:00:45.760361 22871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:00:45.760574 22871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:00:45.765547 22876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:00:45.765599 22883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.765609 22871 server_base.cc:1034] running on GCE node
W20260810 15:00:45.765573 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:00:45.765964 22871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:00:45.766193 22871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:00:45.767328 22871 hybrid_clock.cc:648] HybridClock initialized: now 1786374045767318 us; error 30 us; skew 500 ppm
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:00:45.769784 22871 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:00:45.770236 22871 webserver.cc:460] Webserver started at http://127.28.193.195:36205/ using document root <none> and password file <none>
I20260810 15:00:45.770459 22871 fs_manager.cc:359] Metadata directory not provided
I20260810 15:00:45.770503 22871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:00:45.770630 22871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:00:45.771657 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "508d86c66f3e43b1a9b8599aeef4575c"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "0bf164015fd1cfd841e70e848f03001f"
server_key_iv: "9a55eb769b073092637a780dc835b414"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.771993 22871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "508d86c66f3e43b1a9b8599aeef4575c"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "0bf164015fd1cfd841e70e848f03001f"
server_key_iv: "9a55eb769b073092637a780dc835b414"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.773248 22871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:00:45.773972 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.774122 22871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:00:45.774216 22871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "508d86c66f3e43b1a9b8599aeef4575c"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "0bf164015fd1cfd841e70e848f03001f"
server_key_iv: "9a55eb769b073092637a780dc835b414"
server_key_version: "encryptionkey@0"
I20260810 15:00:45.774281 22871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:00:45.787541 22871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:00:45.789294 22871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:00:45.789426 22871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:00:45.789644 22871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:00:45.789988 22871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:00:45.790024 22871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.790052 22871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:00:45.790076 22871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:00:45.795734 22871 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35067
I20260810 15:00:45.795804 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35067 every 8 connection(s)
I20260810 15:00:45.796104 22871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:00:45.800287 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 22871
I20260810 15:00:45.800383 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:00:45.800575 29447 external_mini_cluster.cc:1428] Setting key 21db4e2b75fbe5f26bcd24aea5292a35
Aug 10 15:00:45 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374045, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:00:45.804607 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.796773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:54393 (local address 127.28.193.254:44961)
0810 15:00:45.796832 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.796834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.797036 (+ 202us) server_negotiation.cc:407] Connection header received
0810 15:00:45.797509 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.797510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.797520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.797558 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:45.798496 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.798985 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.799882 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.800063 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.800122 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.802375 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:45.802377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:45.802377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:45.802384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:45.803620 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.803950 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.803951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.803951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.803995 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.804166 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.804167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.804167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.804258 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:45.804300 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.804490 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.804545 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:00:45.804769 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.796748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:54393)
0810 15:00:45.796918 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.796955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.797415 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.797614 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.797617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.798213 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:00:45.798410 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.798416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.799048 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.799049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.799821 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.799823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.800124 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.800125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.800229 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.800738 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:00:45.800748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:00:45.802303 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:00:45.803684 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.803689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.803691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.803881 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.804039 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:00:45.804039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:00:45.804039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:00:45.804127 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:00:45.804298 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:00:45.804300 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:00:45.804407 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.804480 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.804593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260810 15:00:45.805174 23105 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:00:45.805256 23105 heartbeater.cc:464] Registering TS with master...
I20260810 15:00:45.805425 23105 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:00:45.805799 22300 ts_manager.cc:194] Registered new tserver with Master: 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
I20260810 15:00:45.806141 22300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:54393
I20260810 15:00:45.812426 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:00:45.818984 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.813637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.28.193.254:44961)
0810 15:00:45.813698 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.813700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.813785 (+ 85us) server_negotiation.cc:407] Connection header received
0810 15:00:45.813850 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.813851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.813862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.813922 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:00:45.814415 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.814879 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.815571 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.815773 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.815825 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.816859 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:00:45.816860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:00:45.816861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:00:45.816867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:00:45.817958 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.818252 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.818253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.818253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.818303 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:00:45.818473 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:00:45.818474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:00:45.818474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:00:45.818614 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:00:45.818700 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.818816 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.818898 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:00:45.820310 22300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:00:45.821839 22300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:00:45.827937 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.823765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35067 (local address 127.0.0.1:49862)
0810 15:00:45.824043 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.824053 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.824109 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.824432 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.824435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.824438 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:45.824509 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.824543 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825504 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.825506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.826861 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.826864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.827669 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.827726 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.827736 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.827797 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.827860 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260810 15:00:45.827952 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.824077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.28.193.195:35067)
0810 15:00:45.824237 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.824241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.824252 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:45.824294 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.824296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.824312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.824425 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:45.824652 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825415 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.826944 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.827594 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827672 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.827726 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.827830 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.827874 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260810 15:00:45.828172 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.823954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46119 (local address 127.0.0.1:38040)
0810 15:00:45.824256 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.824265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.824320 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.824593 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.824594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.824597 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:45.824634 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.824635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825967 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.825968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.827065 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.827067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.827938 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.828001 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.828005 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.828053 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.828098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":20,"thread_start_us":47,"threads_started":1}
I20260810 15:00:45.828395 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.824083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.28.193.194:46119)
0810 15:00:45.824236 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.824241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.824311 (+ 70us) server_negotiation.cc:407] Connection header received
0810 15:00:45.824403 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.824405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.824418 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.824485 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:45.824705 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825862 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827132 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.827788 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827847 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.827903 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.828276 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.828318 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260810 15:00:45.828909 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.823761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.0.0.1:38750)
0810 15:00:45.823894 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:45.823937 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:45.823997 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:45.824471 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:45.824472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:45.824474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:45.824539 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.824541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825948 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.825950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.826863 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:45.826866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.828678 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.828679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:45.828739 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.828743 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:45.828782 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 15:00:45.828818 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":73,"thread_start_us":53,"threads_started":1}
I20260810 15:00:45.828790 22985 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 (DEFAULT_TABLE table=test-table [id=57326482099d4e90861137cc1441fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:45.829140 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:45.823874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.28.193.193:43729)
0810 15:00:45.824299 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:45.824302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:45.824311 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:00:45.824350 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:45.824352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:45.824368 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:45.824616 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:45.824686 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.825817 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.827766 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:45.828600 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:45.828941 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:45.829009 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:45.829036 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 15:00:45.829074 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":87,"threads_started":1}
I20260810 15:00:45.829175 22985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a17020ce44e8ebc5d149ba93fb984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:45.829198 22741 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 (DEFAULT_TABLE table=test-table [id=57326482099d4e90861137cc1441fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:45.829519 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a17020ce44e8ebc5d149ba93fb984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:45.829653 22510 tablet_service.cc:1467] Processing CreateTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 (DEFAULT_TABLE table=test-table [id=57326482099d4e90861137cc1441fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:00:45.829995 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5a17020ce44e8ebc5d149ba93fb984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:00:45.832468 23128 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap starting.
I20260810 15:00:45.832975 23133 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Bootstrap starting.
I20260810 15:00:45.833122 23128 tablet_bootstrap.cc:654] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:45.833519 23133 tablet_bootstrap.cc:654] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:45.833560 23128 log.cc:824] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:45.834072 23133 log.cc:824] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:45.834266 23128 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: No bootstrap required, opened a new log
I20260810 15:00:45.834334 23128 ts_tablet_manager.cc:1397] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:00:45.834419 23130 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Bootstrap starting.
I20260810 15:00:45.834867 23133 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: No bootstrap required, opened a new log
I20260810 15:00:45.834990 23133 ts_tablet_manager.cc:1397] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:00:45.835045 23130 tablet_bootstrap.cc:654] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Neither blocks nor log segments found. Creating new log.
I20260810 15:00:45.835623 23130 log.cc:824] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:00:45.835783 23128 raft_consensus.cc:348] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.835985 23128 raft_consensus.cc:374] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:45.836041 23128 raft_consensus.cc:729] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b5ee88abd43808c61196b03fb8b8b, State: Initialized, Role: FOLLOWER
I20260810 15:00:45.836160 23128 consensus_queue.cc:260] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.836400 23130 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: No bootstrap required, opened a new log
I20260810 15:00:45.836452 23128 ts_tablet_manager.cc:1428] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:00:45.836470 23130 ts_tablet_manager.cc:1397] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:00:45.836593 22625 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
I20260810 15:00:45.837304 23133 raft_consensus.cc:348] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.837481 23133 raft_consensus.cc:374] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:45.837517 23133 raft_consensus.cc:729] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f5ffe74d1845f9a1967eb8bb9fc481, State: Initialized, Role: FOLLOWER
I20260810 15:00:45.837630 23133 consensus_queue.cc:260] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.838012 23133 ts_tablet_manager.cc:1428] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 15:00:45.838023 22862 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
I20260810 15:00:45.838022 23130 raft_consensus.cc:348] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.838169 23130 raft_consensus.cc:374] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:00:45.838202 23130 raft_consensus.cc:729] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508d86c66f3e43b1a9b8599aeef4575c, State: Initialized, Role: FOLLOWER
I20260810 15:00:45.838300 23130 consensus_queue.cc:260] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:45.838583 23130 ts_tablet_manager.cc:1428] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:00:45.838595 23105 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
W20260810 15:00:45.891248 22864 tablet.cc:2369] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:45.991192 22626 tablet.cc:2369] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:00:46.046679 23106 tablet.cc:2369] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:00:46.195472 23146 raft_consensus.cc:482] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:00:46.195607 23146 raft_consensus.cc:504] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:46.196002 23146 leader_election.cc:288] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729), 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119)
I20260810 15:00:46.199040 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.196245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.28.193.195:56387)
0810 15:00:46.196291 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:46.196303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:46.196348 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:46.196492 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:46.196494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:46.196496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:46.196563 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:46.196565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.197323 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.197324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:46.198199 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:46.198202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.198779 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:46.198880 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:46.198884 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:46.198934 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:00:46.198966 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260810 15:00:46.199059 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.196306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56387 (local address 127.28.193.193:43729)
0810 15:00:46.196361 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:46.196363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:46.196374 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:00:46.196405 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:46.196406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:46.196416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:46.196470 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:46.196657 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.197241 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198315 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.198708 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198768 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:46.198776 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:46.198948 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 15:00:46.198988 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:00:46.199110 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.196306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:36109 (local address 127.28.193.194:46119)
0810 15:00:46.196378 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:46.196380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:46.196659 (+ 279us) server_negotiation.cc:407] Connection header received
0810 15:00:46.196715 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:46.196716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:46.196723 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:46.196761 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:00:46.196921 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.197427 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198428 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.198808 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198877 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:46.198885 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:46.199015 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 15:00:46.199053 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:00:46.199163 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.196245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46119 (local address 127.28.193.195:36109)
0810 15:00:46.196600 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:46.196609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:00:46.196667 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:00:46.196779 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:00:46.196779 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:00:46.196781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:00:46.196854 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:46.196856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.197526 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.197527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:46.198332 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:00:46.198334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.198883 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.198884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:00:46.198946 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:46.198950 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:00:46.199001 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:00:46.199038 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":162,"thread_start_us":126,"threads_started":1}
I20260810 15:00:46.199450 22545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a17020ce44e8ebc5d149ba93fb984" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5ee88abd43808c61196b03fb8b8b" is_pre_election: true
I20260810 15:00:46.199479 22778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a17020ce44e8ebc5d149ba93fb984" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" is_pre_election: true
I20260810 15:00:46.199605 22545 raft_consensus.cc:2454] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 0.
I20260810 15:00:46.199615 22778 raft_consensus.cc:2454] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 0.
I20260810 15:00:46.199822 22901 leader_election.cc:302] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b, 508d86c66f3e43b1a9b8599aeef4575c; no voters:
I20260810 15:00:46.199951 23146 raft_consensus.cc:2781] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:00:46.200012 23146 raft_consensus.cc:482] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:00:46.200034 23146 raft_consensus.cc:3037] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:46.200827 23146 raft_consensus.cc:504] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:46.200966 23146 leader_election.cc:288] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 election: Requested vote from peers 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729), 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119)
I20260810 15:00:46.201134 22778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a17020ce44e8ebc5d149ba93fb984" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
I20260810 15:00:46.201182 22545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5a17020ce44e8ebc5d149ba93fb984" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5ee88abd43808c61196b03fb8b8b"
I20260810 15:00:46.201239 22778 raft_consensus.cc:3037] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:46.201262 22545 raft_consensus.cc:3037] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:00:46.201994 22545 raft_consensus.cc:2454] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 1.
I20260810 15:00:46.202071 22778 raft_consensus.cc:2454] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 1.
I20260810 15:00:46.202149 22901 leader_election.cc:302] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b, 508d86c66f3e43b1a9b8599aeef4575c; no voters:
I20260810 15:00:46.202278 23146 raft_consensus.cc:2781] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:00:46.202422 23146 raft_consensus.cc:686] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 LEADER]: Becoming Leader. State: Replica: 508d86c66f3e43b1a9b8599aeef4575c, State: Running, Role: LEADER
I20260810 15:00:46.202523 23146 consensus_queue.cc:237] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:00:46.203387 22299 catalog_manager.cc:5495] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c reported cstate change: term changed from 0 to 1, leader changed from <none> to 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195). New cstate: current_term: 1 leader_uuid: "508d86c66f3e43b1a9b8599aeef4575c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } health_report { overall_health: UNKNOWN } } }
I20260810 15:00:46.268070 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.265733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.28.193.195:35067)
0810 15:00:46.265783 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:00:46.265785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:00:46.265794 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:00:46.265837 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:00:46.265838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:00:46.265859 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:00:46.265920 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:00:46.266099 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.266562 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.267362 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:00:46.267593 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:00:46.267644 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:00:46.267767 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:00:46.267881 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:00:46.267939 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:00:46.267952 (+ 13us) server_negotiation.cc:299] Negotiation successful
0810 15:00:46.267969 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:00:46.269944 22778 raft_consensus.cc:1261] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Refusing update from remote peer 508d86c66f3e43b1a9b8599aeef4575c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:46.269970 22545 raft_consensus.cc:1261] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Refusing update from remote peer 508d86c66f3e43b1a9b8599aeef4575c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:00:46.270323 23152 consensus_queue.cc:1035] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:46.270459 23146 consensus_queue.cc:1035] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Connected to new peer: Peer: permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:00:46.272634 23157 mvcc.cc:205] Tried to move back new op lower bound from 7316988093518913536 to 7316988093246033920. Current Snapshot: MvccSnapshot[applied={T|T < 7316988093518913536}]
I20260810 15:00:46.275055 22300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:46.275143 22300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:00:46.275820 22300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Sending DeleteTablet for 3 replicas of tablet ed5a17020ce44e8ebc5d149ba93fb984
I20260810 15:00:46.276163 22985 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:46 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:49862
I20260810 15:00:46.276149 22510 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:46 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:38750
I20260810 15:00:46.276178 22741 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:46 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:38040
I20260810 15:00:46.276504 23169 tablet_replica.cc:330] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: stopping tablet replica
I20260810 15:00:46.276521 23174 tablet_replica.cc:330] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: stopping tablet replica
I20260810 15:00:46.276538 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 22400
I20260810 15:00:46.276614 23174 raft_consensus.cc:2229] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:46.276635 23169 raft_consensus.cc:2229] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:00:46.276723 23174 raft_consensus.cc:2258] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:46.276739 23169 raft_consensus.cc:2258] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:46.279388 23176 tablet_replica.cc:330] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: stopping tablet replica
I20260810 15:00:46.279481 23176 raft_consensus.cc:2229] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:00:46.279561 23176 raft_consensus.cc:2258] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:00:46.282644 23176 ts_tablet_manager.cc:1905] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:46.285332 23176 ts_tablet_manager.cc:1918] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:46.285470 23176 log.cc:1196] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/ed5a17020ce44e8ebc5d149ba93fb984
I20260810 15:00:46.285859 23176 ts_tablet_manager.cc:1939] T ed5a17020ce44e8ebc5d149ba93fb984 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting consensus metadata
I20260810 15:00:46.286432 22276 catalog_manager.cc:4841] TS 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): tablet ed5a17020ce44e8ebc5d149ba93fb984 (table test-table [id=57326482099d4e90861137cc1441fbe3]) successfully deleted
I20260810 15:00:46.287590 23169 ts_tablet_manager.cc:1905] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:00:46.288928 23169 ts_tablet_manager.cc:1918] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:00:46.288998 23169 log.cc:1196] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/ed5a17020ce44e8ebc5d149ba93fb984
I20260810 15:00:46.289237 23169 ts_tablet_manager.cc:1939] T ed5a17020ce44e8ebc5d149ba93fb984 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting consensus metadata
I20260810 15:00:46.289639 22276 catalog_manager.cc:4841] TS 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): tablet ed5a17020ce44e8ebc5d149ba93fb984 (table test-table [id=57326482099d4e90861137cc1441fbe3]) successfully deleted
W20260810 15:00:46.293666 22278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:43729 (error 108)
I20260810 15:00:46.293965 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 22631
I20260810 15:00:46.301973 23119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:00:46.294059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.0.0.1:38766)
0810 15:00:46.301872 (+ 7813us) negotiation.cc:107] Waiting for socket to connect
0810 15:00:46.301934 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7780}
W20260810 15:00:46.327178 22278 catalog_manager.cc:4584] TS 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed5a17020ce44e8ebc5d149ba93fb984: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43729: connect: Connection refused (error 111)
I20260810 15:00:46.331353 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 22871
I20260810 15:00:46.345176 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 22237
I20260810 15:01:06.356701 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--webserver_interface=127.28.193.254
--webserver_port=45975
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:06.438001 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:06.438200 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:06.438230 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:06.439373 23183 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:01:06.439412 23183 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:01:06.439429 23183 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:01:06.439443 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:06.439459 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:06.439476 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:06.439499 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:06.439523 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:06.440999 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:06.441237 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:06.441442 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:06.450501 23202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.450603 23201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.450518 23204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:06.450520 23183 server_base.cc:1034] running on GCE node
I20260810 15:01:06.450999 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:06.451311 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:06.452457 23183 hybrid_clock.cc:648] HybridClock initialized: now 1786374066452449 us; error 31 us; skew 500 ppm
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:06.459565 23183 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:01:06.460067 23183 webserver.cc:460] Webserver started at http://127.28.193.254:45975/ using document root <none> and password file <none>
I20260810 15:01:06.460305 23183 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:06.460372 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:06.461841 23183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.462504 23215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.462654 23183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:06.462738 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "d0318d79c73d4cfc874cf3e254fd8c70"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "941cf05b71b7ca9b43c15be11aae3d02"
server_key_iv: "9b808bc66c5e2754cf1da294aebe19b8"
server_key_version: "encryptionkey@0"
I20260810 15:01:06.463168 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:06.477909 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:06.479956 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:06.480089 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:06.484236 23183 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:44961
I20260810 15:01:06.484292 23312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:44961 every 8 connection(s)
I20260810 15:01:06.484647 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:06.486217 23314 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:01:06.486917 23314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap starting.
I20260810 15:01:06.492606 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 23183
I20260810 15:01:06.492949 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:43729
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=42339
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:01:06.495301 23314 log.cc:824] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:06.497195 23314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:01:06.497476 23314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap complete.
I20260810 15:01:06.499581 23314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:06.499801 23314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Initialized, Role: FOLLOWER
I20260810 15:01:06.499953 23314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:06.500015 23314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:06.500048 23314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:06.500092 23314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:01:06.501068 23314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:06.501157 23314 leader_election.cc:302] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0318d79c73d4cfc874cf3e254fd8c70; no voters:
I20260810 15:01:06.501307 23314 leader_election.cc:288] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:01:06.501362 23323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:01:06.501518 23323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 LEADER]: Becoming Leader. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Running, Role: LEADER
I20260810 15:01:06.501578 23314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:06.501631 23323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:06.501957 23326 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:01:06.501982 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0318d79c73d4cfc874cf3e254fd8c70. Latest consensus state: current_term: 2 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:01:06.502122 23326 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:06.502136 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:06.502503 23337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:06.503304 23337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:06.503412 23337 catalog_manager.cc:1259] Loaded cluster ID: 45c33dc6c84148de89e87b77ed4d3c6f
I20260810 15:01:06.503437 23337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:06.504618 23337 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:06.504807 23337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Loaded TSK: 0
I20260810 15:01:06.513280 23337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Generated new TSK 1
I20260810 15:01:06.513375 23337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:01:06.577327 23319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:06.577536 23319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:06.577569 23319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:06.579301 23319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:06.579355 23319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:06.579393 23319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:06.580859 23319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:43729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=42339
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:06.581110 23319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:06.581329 23319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:06.586380 23366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.586373 23363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.586373 23364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:06.587105 23319 server_base.cc:1034] running on GCE node
I20260810 15:01:06.587307 23319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:06.587531 23319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:06.588685 23319 hybrid_clock.cc:648] HybridClock initialized: now 1786374066588668 us; error 34 us; skew 500 ppm
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:06.591353 23319 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:01:06.591786 23319 webserver.cc:460] Webserver started at http://127.28.193.193:42339/ using document root <none> and password file <none>
I20260810 15:01:06.592046 23319 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:06.592110 23319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:06.593467 23319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.594193 23378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.594359 23319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:01:06.594455 23319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "380b5ee88abd43808c61196b03fb8b8b"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "139da8dfad80d80a3af4e5bbc973ae26"
server_key_iv: "b8ffd31dbb3c58f03d0470d2001a0f5e"
server_key_version: "encryptionkey@0"
I20260810 15:01:06.594880 23319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:06.607142 23319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:06.609598 23319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:06.609787 23319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:06.610220 23319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:06.610831 23398 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:01:06.612780 23319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:01:06.612840 23319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:01:06.612874 23319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:01:06.613785 23319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:01:06.613834 23319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:06.613921 23398 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap starting.
I20260810 15:01:06.620611 23319 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:43729
I20260810 15:01:06.620915 23575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:43729 every 8 connection(s)
I20260810 15:01:06.621040 23319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:06.623641 23398 log.cc:824] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:06.626302 23398 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:01:06.626626 23398 tablet_bootstrap.cc:492] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap complete.
I20260810 15:01:06.627637 23398 ts_tablet_manager.cc:1397] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260810 15:01:06.628609 23398 raft_consensus.cc:348] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:01:06.628855 23398 raft_consensus.cc:729] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b5ee88abd43808c61196b03fb8b8b, State: Initialized, Role: FOLLOWER
I20260810 15:01:06.629020 23398 consensus_queue.cc:260] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } }
I20260810 15:01:06.629139 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 23319
I20260810 15:01:06.629302 23398 ts_tablet_manager.cc:1428] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:01:06.629451 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:46119
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=44439
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:06.635847 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.621942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:32785 (local address 127.28.193.254:44961)
0810 15:01:06.624678 (+ 2736us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.624688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.624733 (+ 45us) server_negotiation.cc:407] Connection header received
0810 15:01:06.624900 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.624907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.624974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.625120 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.626021 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.627358 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.628813 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.630201 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.630314 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.632735 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.632738 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.632739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.632748 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.634228 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.634602 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.634605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.634606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.634689 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.634920 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.634921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.634921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.635168 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.635260 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.635544 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.635661 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2676,"thread_start_us":106,"threads_started":1}
I20260810 15:01:06.635881 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.621913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:32785)
0810 15:01:06.623840 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.623896 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.624459 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.625193 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.625198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.625591 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:06.625896 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.625903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.627470 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.627471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.628718 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.628721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.630295 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.630297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.630433 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.630984 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:06.630993 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:06.632635 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:06.634306 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.634313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.634315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.634527 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.634750 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.634750 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.634751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.634862 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.635250 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:06.635258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:06.635393 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.635483 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.635652 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"thread_start_us":91,"threads_started":1}
I20260810 15:01:06.636478 23576 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:01:06.636565 23576 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:06.636695 23576 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:01:06.637363 23252 ts_manager.cc:194] Registered new tserver with Master: 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729)
I20260810 15:01:06.638445 23252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:32785
I20260810 15:01:06.639433 23576 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.193@KRBTEST.COM
I20260810 15:01:06.645853 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.638152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.28.193.193:43729)
0810 15:01:06.638595 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.638600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.638610 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:06.638644 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.638645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.638660 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.638725 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.639237 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.640193 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.640905 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.641216 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.641274 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.643368 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.643376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.643376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.643382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.644713 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.645102 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.645104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.645104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.645159 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.645341 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.645341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.645342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.645474 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.645530 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.645684 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.645790 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":58,"threads_started":1}
I20260810 15:01:06.645884 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.637931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.0.0.1:42886)
0810 15:01:06.638282 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.638315 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.638392 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.638798 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.638803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.639096 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:06.639148 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.639150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.640261 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.640262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.640822 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.640824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.641288 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.641289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.641362 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.641803 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:06.641808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:06.643286 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:06.644783 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.644789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.644791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.645035 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.645219 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.645220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.645220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.645288 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.645523 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:06.645528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:06.645577 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.645651 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.645781 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":78,"threads_started":1}
I20260810 15:01:06.646246 23475 tablet_service.cc:1514] Processing DeleteTablet for tablet ed5a17020ce44e8ebc5d149ba93fb984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:00:46 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:42886
I20260810 15:01:06.646744 23591 tablet_replica.cc:330] stopping tablet replica
I20260810 15:01:06.646853 23591 raft_consensus.cc:2229] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:06.646919 23591 raft_consensus.cc:2258] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:06.647244 23591 ts_tablet_manager.cc:1905] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:06.649500 23591 ts_tablet_manager.cc:1918] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:06.649576 23591 log.cc:1196] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal/wals/ed5a17020ce44e8ebc5d149ba93fb984
I20260810 15:01:06.649837 23591 ts_tablet_manager.cc:1939] T ed5a17020ce44e8ebc5d149ba93fb984 P 380b5ee88abd43808c61196b03fb8b8b: Deleting consensus metadata
I20260810 15:01:06.650403 23221 catalog_manager.cc:4841] TS 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729): tablet ed5a17020ce44e8ebc5d149ba93fb984 (table test-table [id=57326482099d4e90861137cc1441fbe3]) successfully deleted
W20260810 15:01:06.715734 23586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:06.715939 23586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:06.715961 23586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:06.717026 23586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:06.717058 23586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:06.717082 23586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:06.718554 23586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=44439
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:06.718735 23586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:06.718914 23586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:06.723618 23603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:06.723749 23586 server_base.cc:1034] running on GCE node
W20260810 15:01:06.723649 23599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.723628 23600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:06.724015 23586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:06.724264 23586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:06.725404 23586 hybrid_clock.cc:648] HybridClock initialized: now 1786374066725391 us; error 28 us; skew 500 ppm
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:06.727736 23586 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:01:06.728125 23586 webserver.cc:460] Webserver started at http://127.28.193.194:44439/ using document root <none> and password file <none>
I20260810 15:01:06.728312 23586 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:06.728368 23586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:06.729748 23586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:06.730394 23617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.730563 23586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.730651 23586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "b6cbabf710240d185d1b2a4f59853b16"
server_key_iv: "8ae54c31d525a0933487c57958b4d34e"
server_key_version: "encryptionkey@0"
I20260810 15:01:06.731003 23586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:06.756395 23586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:06.758533 23586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:06.758664 23586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:06.758925 23586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:06.759243 23586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:06.759279 23586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.759302 23586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:06.759315 23586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.764720 23586 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:46119
I20260810 15:01:06.764778 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:46119 every 8 connection(s)
I20260810 15:01:06.765103 23586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:01:06.767037 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 23586
I20260810 15:01:06.767359 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:35067
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=36205
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:06.774667 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.765799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:58933 (local address 127.28.193.254:44961)
0810 15:01:06.765853 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.765855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.766102 (+ 247us) server_negotiation.cc:407] Connection header received
0810 15:01:06.766565 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.766566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.766579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.766630 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.767816 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.768314 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.769504 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.769832 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.769907 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.771883 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.771885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.771886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.771895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.773378 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.773761 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.773763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.773763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.773827 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.774046 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.774047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.774047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.774182 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.774250 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.774479 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.774572 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:01:06.774711 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.765798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:58933)
0810 15:01:06.765972 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.766018 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.766487 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.766702 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.766705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.767451 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:06.767716 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.767723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.768411 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.768412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.769417 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.769420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.769922 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.769923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.770027 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.770463 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:06.770470 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:06.771780 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:06.773462 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.773468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.773470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.773675 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.773894 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.773894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.773895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.773989 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.774249 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:06.774252 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:06.774365 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.774438 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.774551 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260810 15:01:06.775182 23803 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:01:06.775270 23803 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:06.775449 23803 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:01:06.775852 23252 ts_manager.cc:194] Registered new tserver with Master: 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119)
I20260810 15:01:06.776331 23252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:58933
W20260810 15:01:06.852600 23808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:06.852869 23808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:06.852903 23808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:06.854503 23808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:06.854554 23808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:06.854580 23808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:06.856066 23808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:35067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=36205
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:06.856304 23808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:06.856530 23808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:01:06.861575 23808 server_base.cc:1034] running on GCE node
W20260810 15:01:06.861549 23819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.861531 23817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:06.861541 23822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:06.861995 23808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:06.862262 23808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:06.863420 23808 hybrid_clock.cc:648] HybridClock initialized: now 1786374066863398 us; error 32 us; skew 500 ppm
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:06.866024 23808 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:01:06.866528 23808 webserver.cc:460] Webserver started at http://127.28.193.195:36205/ using document root <none> and password file <none>
I20260810 15:01:06.866791 23808 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:06.866856 23808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:06.868261 23808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.868912 23832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.869064 23808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.869172 23808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "508d86c66f3e43b1a9b8599aeef4575c"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "0bf164015fd1cfd841e70e848f03001f"
server_key_iv: "9a55eb769b073092637a780dc835b414"
server_key_version: "encryptionkey@0"
I20260810 15:01:06.869517 23808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:06.893859 23808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:06.895844 23808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:06.895964 23808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:06.896194 23808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:06.896510 23808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:06.896545 23808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.896575 23808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:06.896593 23808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:06.902534 23808 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35067
I20260810 15:01:06.902628 24038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35067 every 8 connection(s)
I20260810 15:01:06.902952 23808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:06.903457 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 23808
Aug 10 15:01:06 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374066, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:06.910610 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.903819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.28.193.254:44961)
0810 15:01:06.903922 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.903924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.904033 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:01:06.904105 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.904106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.904116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.904169 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.904885 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.905386 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.906223 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.906445 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.906496 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.907821 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.907822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.907822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.907827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.909388 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.909750 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.909751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.909752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.909821 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.910054 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.910055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.910055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.910173 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.910352 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.910422 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.910513 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260810 15:01:06.912604 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.903654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:54263 (local address 127.28.193.254:44961)
0810 15:01:06.903710 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.903712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.903976 (+ 264us) server_negotiation.cc:407] Connection header received
0810 15:01:06.904625 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.904626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.904631 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.904659 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.905688 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.906151 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.907356 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.907538 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.907589 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.909901 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.909903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.909903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.909911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.911332 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.911682 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.911683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.911683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.911732 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.911930 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.911931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.911932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.912049 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.912110 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.912427 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.912512 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:01:06.913026 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.903669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:54263)
0810 15:01:06.903830 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.903896 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.904536 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.904760 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.904764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.905362 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:06.905606 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.905612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.906235 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.906236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.907267 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.907270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.907611 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.907612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.907748 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.908195 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:06.908205 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:06.909823 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:06.911412 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.911418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.911421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.911625 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.911772 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:06.911772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:06.911773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:06.911854 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:06.912117 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:06.912123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:06.912284 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.912391 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.912574 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260810 15:01:06.913463 24039 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44961
I20260810 15:01:06.913555 24039 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:06.913750 24039 heartbeater.cc:511] Master 127.28.193.254:44961 requested a full tablet report, sending...
I20260810 15:01:06.914108 23252 ts_manager.cc:194] Registered new tserver with Master: 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
I20260810 15:01:06.914456 23252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:54263
I20260810 15:01:06.922173 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:06.924850 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.922531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.28.193.254:44961)
0810 15:01:06.922591 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.922593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.922653 (+ 60us) server_negotiation.cc:407] Connection header received
0810 15:01:06.922739 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.922740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.922783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.922837 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:06.922978 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.923453 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.924184 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.924378 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.924440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.924589 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:06.924641 (+ 52us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 15:01:06.924784 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40918: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 15:01:06.924911 23582 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40918: token expired
W20260810 15:01:06.925822 23115 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.193.254:44961)
I20260810 15:01:06.931563 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.926068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.28.193.254:44961)
0810 15:01:06.926106 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.926108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.926115 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:01:06.926162 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.926163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.926172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.926216 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:06.926668 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.927110 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.927760 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.927954 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.928004 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.929151 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:06.929152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:06.929152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:06.929158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:06.930485 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.930836 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.930837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.930837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.930925 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:06.931123 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:06.931124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:06.931124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:06.931223 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:06.931298 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.931436 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.931499 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:01:06.933013 23252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:06.934767 23252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:06.937188 23475 tablet_service.cc:1467] Processing CreateTablet for tablet e7ab0462791346cb98602ca9d159ba31 (DEFAULT_TABLE table=test-table [id=f544efdbead0412aa6e05cc8a32024cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:06.937431 23475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ab0462791346cb98602ca9d159ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:06.939018 23398 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap starting.
I20260810 15:01:06.939320 23398 tablet_bootstrap.cc:654] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:06.939853 23398 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: No bootstrap required, opened a new log
I20260810 15:01:06.939894 23398 ts_tablet_manager.cc:1397] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.940224 23398 raft_consensus.cc:348] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.940310 23398 raft_consensus.cc:374] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:06.940358 23398 raft_consensus.cc:729] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b5ee88abd43808c61196b03fb8b8b, State: Initialized, Role: FOLLOWER
I20260810 15:01:06.940402 23398 consensus_queue.cc:260] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.940485 23398 ts_tablet_manager.cc:1428] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:06.940945 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.936793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46119 (local address 127.0.0.1:33936)
0810 15:01:06.936847 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.936858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.936899 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.937422 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.937423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.937426 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:06.937479 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.937481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.938901 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.938902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.939917 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.939919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.940674 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.940675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.940733 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.940746 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.940779 (+ 33us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.940826 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:01:06.941087 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.936898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.28.193.194:46119)
0810 15:01:06.937236 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.937241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.937251 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:06.937294 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.937296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.937312 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.937675 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:06.937795 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.938812 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.939988 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.940587 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.940872 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.940947 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.940970 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.941005 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":98,"threads_started":1}
I20260810 15:01:06.941174 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.936892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35067 (local address 127.0.0.1:49600)
0810 15:01:06.937066 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:06.937074 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:06.937119 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:06.937578 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:06.937580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:06.937582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:06.937628 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.937629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.938566 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.938567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.939734 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:06.939737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.940658 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.940659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:06.940723 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.940750 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 15:01:06.940796 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:01:06.940826 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260810 15:01:06.941391 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:06.937112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.28.193.195:35067)
0810 15:01:06.937267 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:06.937271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:06.937281 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:06.937324 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:06.937326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:06.937342 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:06.937452 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:06.937709 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.938487 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.939816 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:06.940570 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:06.940652 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:06.940716 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:06.940916 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 15:01:06.940964 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260810 15:01:06.941841 23688 tablet_service.cc:1467] Processing CreateTablet for tablet e7ab0462791346cb98602ca9d159ba31 (DEFAULT_TABLE table=test-table [id=f544efdbead0412aa6e05cc8a32024cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:06.942099 23918 tablet_service.cc:1467] Processing CreateTablet for tablet e7ab0462791346cb98602ca9d159ba31 (DEFAULT_TABLE table=test-table [id=f544efdbead0412aa6e05cc8a32024cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:06.942219 23688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ab0462791346cb98602ca9d159ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:06.942421 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7ab0462791346cb98602ca9d159ba31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:06.944872 24057 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Bootstrap starting.
I20260810 15:01:06.945101 24058 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Bootstrap starting.
I20260810 15:01:06.945528 24057 tablet_bootstrap.cc:654] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:06.945636 24058 tablet_bootstrap.cc:654] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:06.946030 24057 log.cc:824] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:06.946127 24058 log.cc:824] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:06.946729 24057 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: No bootstrap required, opened a new log
I20260810 15:01:06.946784 24057 ts_tablet_manager.cc:1397] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:01:06.946776 24058 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: No bootstrap required, opened a new log
I20260810 15:01:06.946833 24058 ts_tablet_manager.cc:1397] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:06.948041 24057 raft_consensus.cc:348] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.948177 24057 raft_consensus.cc:374] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:06.948065 24058 raft_consensus.cc:348] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.948218 24057 raft_consensus.cc:729] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f5ffe74d1845f9a1967eb8bb9fc481, State: Initialized, Role: FOLLOWER
I20260810 15:01:06.948221 24058 raft_consensus.cc:374] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:06.948269 24058 raft_consensus.cc:729] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508d86c66f3e43b1a9b8599aeef4575c, State: Initialized, Role: FOLLOWER
I20260810 15:01:06.948333 24057 consensus_queue.cc:260] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.948395 24058 consensus_queue.cc:260] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:06.948650 23803 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
I20260810 15:01:06.948662 24058 ts_tablet_manager.cc:1428] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:06.948683 24039 heartbeater.cc:503] Master 127.28.193.254:44961 was elected leader, sending a full tablet report...
I20260810 15:01:06.948868 24057 ts_tablet_manager.cc:1428] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260810 15:01:07.015625 23804 tablet.cc:2369] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:07.093159 24064 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:07.093291 24064 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:07.093698 24064 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729)
I20260810 15:01:07.096536 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.094031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:41197 (local address 127.28.193.194:46119)
0810 15:01:07.094077 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:07.094079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:07.094232 (+ 153us) server_negotiation.cc:407] Connection header received
0810 15:01:07.094308 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:07.094310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:07.094319 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:07.094383 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:07.094513 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.094959 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.095838 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.096209 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.096265 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:07.096272 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:07.096421 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 15:01:07.096464 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:01:07.096558 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.093939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.28.193.195:34401)
0810 15:01:07.093979 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:07.093990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:07.094038 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:07.094239 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:07.094240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:07.094242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:07.094316 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:07.094318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.095019 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.095020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:07.095749 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:07.095751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.096322 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.096323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:07.096388 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:07.096391 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:07.096444 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:01:07.096478 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 15:01:07.096558 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.094007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:34401 (local address 127.28.193.193:43729)
0810 15:01:07.094064 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:07.094065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:07.094075 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:07.094107 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:07.094107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:07.094140 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:07.094223 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:07.094395 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.094937 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.095840 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.096238 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.096308 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:07.096384 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:07.096455 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 15:01:07.096493 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:01:07.096567 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.093940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46119 (local address 127.28.193.195:41197)
0810 15:01:07.094173 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:07.094183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:07.094247 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:07.094392 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:07.094393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:07.094395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:07.094458 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:07.094459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.095036 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.095036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:07.095753 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:07.095755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.096291 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.096292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:07.096354 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:07.096358 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:07.096409 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:01:07.096448 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":54,"thread_start_us":119,"threads_started":1}
I20260810 15:01:07.096838 23500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ab0462791346cb98602ca9d159ba31" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5ee88abd43808c61196b03fb8b8b" is_pre_election: true
I20260810 15:01:07.097003 23500 raft_consensus.cc:2454] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 0.
I20260810 15:01:07.097103 23722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ab0462791346cb98602ca9d159ba31" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" is_pre_election: true
I20260810 15:01:07.097250 23722 raft_consensus.cc:2454] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 0.
I20260810 15:01:07.097250 23836 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b, 508d86c66f3e43b1a9b8599aeef4575c; no voters:
I20260810 15:01:07.097368 24064 raft_consensus.cc:2781] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:07.097438 24064 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:07.097462 24064 raft_consensus.cc:3037] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:07.098187 24064 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:07.098330 24064 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 election: Requested vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729)
I20260810 15:01:07.098589 23722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ab0462791346cb98602ca9d159ba31" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
I20260810 15:01:07.098619 23500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7ab0462791346cb98602ca9d159ba31" candidate_uuid: "508d86c66f3e43b1a9b8599aeef4575c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5ee88abd43808c61196b03fb8b8b"
I20260810 15:01:07.098670 23722 raft_consensus.cc:3037] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:07.098707 23500 raft_consensus.cc:3037] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:07.099402 23722 raft_consensus.cc:2454] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 1.
I20260810 15:01:07.099475 23500 raft_consensus.cc:2454] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508d86c66f3e43b1a9b8599aeef4575c in term 1.
I20260810 15:01:07.099591 23835 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508d86c66f3e43b1a9b8599aeef4575c, 73f5ffe74d1845f9a1967eb8bb9fc481; no voters:
I20260810 15:01:07.099705 24064 raft_consensus.cc:2781] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:07.099852 24064 raft_consensus.cc:686] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 LEADER]: Becoming Leader. State: Replica: 508d86c66f3e43b1a9b8599aeef4575c, State: Running, Role: LEADER
I20260810 15:01:07.099949 24064 consensus_queue.cc:237] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:07.100857 23248 catalog_manager.cc:5495] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c reported cstate change: term changed from 0 to 1, leader changed from <none> to 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195). New cstate: current_term: 1 leader_uuid: "508d86c66f3e43b1a9b8599aeef4575c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:07.121007 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.118923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49604 (local address 127.28.193.195:35067)
0810 15:01:07.118966 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:07.118968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:07.118978 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:07.119035 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:07.119036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:07.119052 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:07.119108 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:07.119188 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.119630 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.120340 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:07.120546 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:07.120607 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:07.120737 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:07.120840 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:07.120897 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:07.120910 (+ 13us) server_negotiation.cc:299] Negotiation successful
0810 15:01:07.120925 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:01:07.122011 23578 tablet.cc:2369] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:07.123035 23722 raft_consensus.cc:1261] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Refusing update from remote peer 508d86c66f3e43b1a9b8599aeef4575c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:07.123035 23500 raft_consensus.cc:1261] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Refusing update from remote peer 508d86c66f3e43b1a9b8599aeef4575c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:07.123437 24066 consensus_queue.cc:1035] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:07.123564 24064 consensus_queue.cc:1035] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [LEADER]: Connected to new peer: Peer: permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:07.125375 24068 mvcc.cc:205] Tried to move back new op lower bound from 7316988178932932608 to 7316988178842066944. Current Snapshot: MvccSnapshot[applied={T|T < 7316988178932932608}]
I20260810 15:01:07.125521 24070 mvcc.cc:205] Tried to move back new op lower bound from 7316988178932932608 to 7316988178842066944. Current Snapshot: MvccSnapshot[applied={T|T < 7316988178932932608}]
I20260810 15:01:07.127835 23248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:07.127929 23248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:07.128603 23248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Sending DeleteTablet for 3 replicas of tablet e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:07.128906 23475 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ab0462791346cb98602ca9d159ba31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:07 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:42886
I20260810 15:01:07.128904 23918 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ab0462791346cb98602ca9d159ba31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:07 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:49600
I20260810 15:01:07.128888 23688 tablet_service.cc:1514] Processing DeleteTablet for tablet e7ab0462791346cb98602ca9d159ba31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:07 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:33936
I20260810 15:01:07.129007 23591 tablet_replica.cc:330] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: stopping tablet replica
I20260810 15:01:07.129040 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 23319
I20260810 15:01:07.129058 23591 raft_consensus.cc:2229] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:07.129094 23591 raft_consensus.cc:2258] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:07.129235 24082 tablet_replica.cc:330] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: stopping tablet replica
I20260810 15:01:07.129349 24084 tablet_replica.cc:330] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: stopping tablet replica
I20260810 15:01:07.129459 24084 raft_consensus.cc:2229] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:01:07.129599 24082 raft_consensus.cc:2229] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:07.129781 24084 raft_consensus.cc:2258] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:07.129958 24082 raft_consensus.cc:2258] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:07.136965 24082 ts_tablet_manager.cc:1905] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:07.137282 24084 ts_tablet_manager.cc:1905] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:07.138733 24082 ts_tablet_manager.cc:1918] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:07.138820 24082 log.cc:1196] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:07.138876 24084 ts_tablet_manager.cc:1918] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:07.138934 24084 log.cc:1196] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:07.139106 24082 ts_tablet_manager.cc:1939] T e7ab0462791346cb98602ca9d159ba31 P 73f5ffe74d1845f9a1967eb8bb9fc481: Deleting consensus metadata
I20260810 15:01:07.139169 24084 ts_tablet_manager.cc:1939] T e7ab0462791346cb98602ca9d159ba31 P 508d86c66f3e43b1a9b8599aeef4575c: Deleting consensus metadata
I20260810 15:01:07.139540 23219 catalog_manager.cc:4841] TS 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): tablet e7ab0462791346cb98602ca9d159ba31 (table test-table [id=f544efdbead0412aa6e05cc8a32024cc]) successfully deleted
I20260810 15:01:07.139631 23219 catalog_manager.cc:4841] TS 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): tablet e7ab0462791346cb98602ca9d159ba31 (table test-table [id=f544efdbead0412aa6e05cc8a32024cc]) successfully deleted
W20260810 15:01:07.144194 23221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:43729 (error 108)
I20260810 15:01:07.144502 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 23586
I20260810 15:01:07.151361 24051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:07.144566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:43729 (local address 127.0.0.1:42900)
0810 15:01:07.151256 (+ 6690us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:07.151320 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6663}
W20260810 15:01:07.151513 23221 catalog_manager.cc:4584] TS 380b5ee88abd43808c61196b03fb8b8b (127.28.193.193:43729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e7ab0462791346cb98602ca9d159ba31: Network error: Client connection negotiation failed: client connection to 127.28.193.193:43729: connect: Connection refused (error 111)
I20260810 15:01:07.159646 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 23808
I20260810 15:01:07.171988 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 23183
I20260810 15:01:27.185105 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--webserver_interface=127.28.193.254
--webserver_port=45975
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:27.271435 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:27.271637 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:27.271658 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:27.272727 24087 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:01:27.272758 24087 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:01:27.272769 24087 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:01:27.272779 24087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:27.272790 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:27.272802 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:27.272819 24087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:27.272830 24087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:27.274277 24087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=45975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:27.274474 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:27.274639 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:27.279497 24098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.279486 24094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.279503 24087 server_base.cc:1034] running on GCE node
W20260810 15:01:27.279486 24093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.279928 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:27.280213 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:27.281381 24087 hybrid_clock.cc:648] HybridClock initialized: now 1786374087281366 us; error 29 us; skew 500 ppm
Aug 10 15:01:27 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:27.287001 24087 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:01:27.287428 24087 webserver.cc:460] Webserver started at http://127.28.193.254:45975/ using document root <none> and password file <none>
I20260810 15:01:27.287626 24087 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:27.287674 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:27.288939 24087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:01:27.289543 24111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.289703 24087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.289824 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "d0318d79c73d4cfc874cf3e254fd8c70"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "941cf05b71b7ca9b43c15be11aae3d02"
server_key_iv: "9b808bc66c5e2754cf1da294aebe19b8"
server_key_version: "encryptionkey@0"
I20260810 15:01:27.290160 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:27.308669 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:27.310603 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:27.310724 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:27.314744 24087 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:44961
I20260810 15:01:27.314782 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:44961 every 8 connection(s)
I20260810 15:01:27.315142 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:27.316613 24204 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:01:27.317265 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap starting.
I20260810 15:01:27.321103 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 24087
I20260810 15:01:27.321367 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:43729
--local_ip_for_outbound_sockets=127.28.193.193
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=42339
--webserver_interface=127.28.193.193
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:01:27.325311 24204 log.cc:824] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:27.327749 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:01:27.328045 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: Bootstrap complete.
I20260810 15:01:27.330195 24204 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:27.330425 24204 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Initialized, Role: FOLLOWER
I20260810 15:01:27.330591 24204 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:27.330673 24204 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:27.330724 24204 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:27.330771 24204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:01:27.331758 24204 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:27.331856 24204 leader_election.cc:302] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0318d79c73d4cfc874cf3e254fd8c70; no voters:
I20260810 15:01:27.332024 24204 leader_election.cc:288] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:01:27.332074 24211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:01:27.332192 24211 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [term 3 LEADER]: Becoming Leader. State: Replica: d0318d79c73d4cfc874cf3e254fd8c70, State: Running, Role: LEADER
I20260810 15:01:27.332297 24204 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:27.332279 24211 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } }
I20260810 15:01:27.332613 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:01:27.332697 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:27.333022 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0318d79c73d4cfc874cf3e254fd8c70. Latest consensus state: current_term: 3 leader_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0318d79c73d4cfc874cf3e254fd8c70" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44961 } } }
I20260810 15:01:27.333096 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:27.334039 24238 catalog_manager.cc:1259] Loaded cluster ID: 45c33dc6c84148de89e87b77ed4d3c6f
I20260810 15:01:27.334080 24238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: loading cluster ID for follower catalog manager: success
I20260810 15:01:27.334865 24238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0318d79c73d4cfc874cf3e254fd8c70: acquiring CA information for follower catalog manager: success
F20260810 15:01:27.335196 24238 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9335589ad google::LogMessage::Fail() at ??:0
@ 0x7fa93355b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9335584de google::LogMessage::Flush() at ??:0
@ 0x7fa933559a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa935cc5104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa93b4b56b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fa93b4b5989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fa93b4b5e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fa93b4b88c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fa93541b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa93627f6db start_thread at ??:0
@ 0x7fa93205661f clone at ??:0
W20260810 15:01:27.436545 24208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:27.436744 24208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:27.436766 24208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:27.437934 24208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:27.437973 24208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:27.437999 24208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:27.439424 24208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:43729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=42339
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:27.439636 24208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:27.439858 24208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:27.444593 24266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.444626 24268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.444582 24265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.445109 24208 server_base.cc:1034] running on GCE node
I20260810 15:01:27.445353 24208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:27.445636 24208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:27.446777 24208 hybrid_clock.cc:648] HybridClock initialized: now 1786374087446762 us; error 34 us; skew 500 ppm
Aug 10 15:01:27 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:27.449226 24208 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:01:27.449653 24208 webserver.cc:460] Webserver started at http://127.28.193.193:42339/ using document root <none> and password file <none>
I20260810 15:01:27.449901 24208 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:27.449955 24208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:27.451220 24208 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:27.451844 24279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.452049 24208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:27.452149 24208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "380b5ee88abd43808c61196b03fb8b8b"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "139da8dfad80d80a3af4e5bbc973ae26"
server_key_iv: "b8ffd31dbb3c58f03d0470d2001a0f5e"
server_key_version: "encryptionkey@0"
I20260810 15:01:27.452600 24208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:27.478988 24208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:27.481354 24208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:27.481516 24208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:27.481809 24208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:27.482400 24290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:01:27.483917 24208 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:01:27.483978 24208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:01:27.484014 24208 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:01:27.484956 24208 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:01:27.485004 24208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:01:27.485116 24290 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap starting.
I20260810 15:01:27.492264 24208 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:43729
I20260810 15:01:27.492738 24208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:27.495920 24290 log.cc:824] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:27.496392 24491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:43729 every 8 connection(s)
I20260810 15:01:27.497527 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 24208
I20260810 15:01:27.497840 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:46119
--local_ip_for_outbound_sockets=127.28.193.194
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=44439
--webserver_interface=127.28.193.194
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:01:27.498775 24290 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:01:27.499082 24290 tablet_bootstrap.cc:492] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Bootstrap complete.
I20260810 15:01:27.500183 24290 ts_tablet_manager.cc:1397] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260810 15:01:27.501607 24290 raft_consensus.cc:348] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:27.501979 24290 raft_consensus.cc:729] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b5ee88abd43808c61196b03fb8b8b, State: Initialized, Role: FOLLOWER
I20260810 15:01:27.502132 24290 consensus_queue.cc:260] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:27.502442 24290 ts_tablet_manager.cc:1428] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260810 15:01:27.551152 24495 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:01:27.493535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:58203)
0810 15:01:27.493778 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.493844 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:27.494526 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:27.550904 (+ 56378us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
W20260810 15:01:27.551431 24492 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44961 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:44961: Client connection negotiation failed: client connection to 127.28.193.254:44961: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:01:27.551774 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.551654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:46953)
0810 15:01:27.551706 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.551749 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:01:27.552102 24495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.552020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:46247)
0810 15:01:27.552057 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.552083 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:01:27.552253 24492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:01:27.583210 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:27.583400 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:27.583422 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:27.584560 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:27.584601 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:27.584625 24498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:27.586125 24498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=44439
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:27.586344 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:27.586571 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:27.591461 24516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.591472 24511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.591478 24513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.591784 24498 server_base.cc:1034] running on GCE node
I20260810 15:01:27.591982 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:27.592175 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:27.593315 24498 hybrid_clock.cc:648] HybridClock initialized: now 1786374087593301 us; error 34 us; skew 500 ppm
Aug 10 15:01:27 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:27.595798 24498 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:01:27.596243 24498 webserver.cc:460] Webserver started at http://127.28.193.194:44439/ using document root <none> and password file <none>
I20260810 15:01:27.596519 24498 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:27.596618 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:27.597998 24498 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:27.598613 24529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.598775 24498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.598850 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "73f5ffe74d1845f9a1967eb8bb9fc481"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "b6cbabf710240d185d1b2a4f59853b16"
server_key_iv: "8ae54c31d525a0933487c57958b4d34e"
server_key_version: "encryptionkey@0"
I20260810 15:01:27.599193 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:27.607308 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:27.609354 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:27.609493 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:27.609760 24498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:27.610074 24498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:27.610109 24498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.610136 24498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:27.610159 24498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.615974 24498 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:46119
I20260810 15:01:27.616017 24716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:46119 every 8 connection(s)
I20260810 15:01:27.616354 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:01:27.617389 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.617013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:36669)
0810 15:01:27.617166 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.617238 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260810 15:01:27.617765 24718 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44961 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:44961: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
I20260810 15:01:27.618044 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.617941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:48793)
0810 15:01:27.617990 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.618022 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:01:27.618042 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 24498
I20260810 15:01:27.618333 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:35067
--local_ip_for_outbound_sockets=127.28.193.195
--tserver_master_addrs=127.28.193.254:44961
--webserver_port=36205
--webserver_interface=127.28.193.195
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:01:27.618350 24722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.618260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:54797)
0810 15:01:27.618304 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.618329 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:01:27.618561 24718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:01:27.699818 24727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:27.700021 24727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:27.700043 24727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:27.701119 24727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:27.701150 24727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:27.701174 24727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:27.702661 24727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:35067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=36205
--tserver_master_addrs=127.28.193.254:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:27.702869 24727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:27.703050 24727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:27.707809 24742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:27.707861 24740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.708016 24727 server_base.cc:1034] running on GCE node
W20260810 15:01:27.708016 24736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:27.708268 24727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:27.708460 24727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:27.709601 24727 hybrid_clock.cc:648] HybridClock initialized: now 1786374087709589 us; error 29 us; skew 500 ppm
Aug 10 15:01:27 dist-test-slave-cvwc krb5kdc[22211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:27.712061 24727 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:01:27.712471 24727 webserver.cc:460] Webserver started at http://127.28.193.195:36205/ using document root <none> and password file <none>
I20260810 15:01:27.712652 24727 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:27.712692 24727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:27.714002 24727 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:27.714556 24755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.714706 24727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:27.714788 24727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "508d86c66f3e43b1a9b8599aeef4575c"
format_stamp: "Formatted at 2026-08-10 15:00:45 on dist-test-slave-cvwc"
server_key: "0bf164015fd1cfd841e70e848f03001f"
server_key_iv: "9a55eb769b073092637a780dc835b414"
server_key_version: "encryptionkey@0"
I20260810 15:01:27.715116 24727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:27.721324 24727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:27.722962 24727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:27.723083 24727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:27.723272 24727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:27.723573 24727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:27.723613 24727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.723640 24727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:27.723664 24727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:27.729220 24727 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35067
I20260810 15:01:27.729278 24950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35067 every 8 connection(s)
I20260810 15:01:27.729596 24727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:27.730577 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.730227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:40659)
0810 15:01:27.730374 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.730443 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260810 15:01:27.730955 24951 heartbeater.cc:644] Failed to heartbeat to 127.28.193.254:44961 (0 consecutive failures): Network error: Failed to ping master at 127.28.193.254:44961: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
I20260810 15:01:27.731221 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.731138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:54835)
0810 15:01:27.731177 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.731204 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:01:27.731503 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:27.731426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:58365)
0810 15:01:27.731462 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:27.731487 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:01:27.731593 24951 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:01:27.733681 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 24727
I20260810 15:01:28.552845 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:28.552608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:40031)
0810 15:01:28.552730 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:28.552771 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260810 15:01:28.619043 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:28.618845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:59007)
0810 15:01:28.618955 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:28.618990 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260810 15:01:28.732132 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:28.731895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:55679)
0810 15:01:28.732028 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:28.732067 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260810 15:01:29.012315 24962 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:29.012450 24962 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:29.012933 24962 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
Aug 10 15:01:29 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.194@KRBTEST.COM
Aug 10 15:01:29 dist-test-slave-cvwc krb5kdc[22211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374087, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.195@KRBTEST.COM
I20260810 15:01:29.023195 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.013205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35067 (local address 127.28.193.193:54415)
0810 15:01:29.013324 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:29.013333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:29.013376 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:29.014201 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:29.014202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:29.014582 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:29.014771 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:29.014777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.015844 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.015845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:29.016618 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:29.016620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.017213 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.017214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:29.017287 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:29.017898 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:29.017903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:29.019818 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:29.021772 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:29.021773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:29.021773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:29.021952 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:29.022194 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:29.022195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:29.022195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:29.022276 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:29.022675 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:29.022676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:29.022780 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 15:01:29.022894 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 15:01:29.023078 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":82,"thread_start_us":53,"threads_started":1}
I20260810 15:01:29.023212 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.013333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:54415 (local address 127.28.193.195:35067)
0810 15:01:29.013467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:29.013476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:29.013507 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:01:29.014035 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:29.014038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:29.014078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:29.014195 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:29.014885 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.015733 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.016704 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.017138 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.017252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:29.019915 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:29.019919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:29.019919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:29.019930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:29.021653 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:29.022023 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:29.022026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:29.022026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:29.022129 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:29.022342 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:29.022343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:29.022343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:29.022606 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:29.022683 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:29.022961 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 15:01:29.023102 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260810 15:01:29.023247 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.013086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:46119 (local address 127.28.193.193:35275)
0810 15:01:29.013141 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:29.013155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:29.013236 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:29.014184 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:29.014187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:29.014583 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:29.014771 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:29.014776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.015823 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.015824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:29.016616 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:29.016618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.017168 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.017169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:29.017274 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:29.017868 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:29.017876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:29.019447 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:29.021520 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:29.021526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:29.021528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:29.021788 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:29.022049 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:29.022050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:29.022050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:29.022155 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:29.022620 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:29.022626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:29.022780 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 15:01:29.022894 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 15:01:29.023139 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":130}
I20260810 15:01:29.023605 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.013219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:35275 (local address 127.28.193.194:46119)
0810 15:01:29.013418 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:29.013431 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:29.013466 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:01:29.013965 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:29.013968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:29.014013 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:29.014143 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:29.014898 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.015726 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.016698 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:29.017090 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:29.017187 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:29.019552 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:29.019557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:29.019557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:29.019570 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:29.021423 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:29.021877 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:29.021880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:29.021881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:29.021974 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:29.022235 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:29.022236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:29.022236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:29.022544 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:29.022643 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:29.023350 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 15:01:29.023498 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":76,"threads_started":1}
W20260810 15:01:29.024269 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
W20260810 15:01:29.024360 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:29.024394 24282 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b; no voters: 508d86c66f3e43b1a9b8599aeef4575c, 73f5ffe74d1845f9a1967eb8bb9fc481
I20260810 15:01:29.024518 24962 raft_consensus.cc:2726] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:01:29.553557 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.553310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:40501)
0810 15:01:29.553465 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:29.553508 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260810 15:01:29.619781 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.619537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:54081)
0810 15:01:29.619681 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:29.619721 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260810 15:01:29.732759 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:29.732554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:41193)
0810 15:01:29.732676 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:29.732719 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260810 15:01:30.554325 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:30.554089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:43877)
0810 15:01:30.554217 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:30.554272 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260810 15:01:30.620560 24979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:30.620304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:45905)
0810 15:01:30.620452 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:30.620504 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260810 15:01:30.733465 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:30.733197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:39571)
0810 15:01:30.733379 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:30.733421 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260810 15:01:30.915723 24983 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:30.915827 24983 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:30.916014 24983 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
W20260810 15:01:30.916523 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
W20260810 15:01:30.916595 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:30.916657 24282 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b; no voters: 508d86c66f3e43b1a9b8599aeef4575c, 73f5ffe74d1845f9a1967eb8bb9fc481
I20260810 15:01:30.916743 24983 raft_consensus.cc:2726] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:01:31.555017 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:31.554803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:41069)
0810 15:01:31.554915 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:31.554967 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260810 15:01:31.621162 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:31.620965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:56621)
0810 15:01:31.621085 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:31.621122 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260810 15:01:31.734045 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:31.733851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:52785)
0810 15:01:31.733963 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:31.733995 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260810 15:01:32.445685 24993 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:32.445847 24993 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:32.446015 24993 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
W20260810 15:01:32.446426 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
W20260810 15:01:32.446512 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:32.446535 24282 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b; no voters: 508d86c66f3e43b1a9b8599aeef4575c, 73f5ffe74d1845f9a1967eb8bb9fc481
I20260810 15:01:32.446630 24993 raft_consensus.cc:2726] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:01:32.555738 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:32.555517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:33185)
0810 15:01:32.555650 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:32.555691 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260810 15:01:32.621816 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:32.621593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:34349)
0810 15:01:32.621741 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:32.621775 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260810 15:01:32.734673 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:32.734467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:60093)
0810 15:01:32.734596 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:32.734631 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260810 15:01:33.556355 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:33.556154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:38581)
0810 15:01:33.556270 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:33.556299 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260810 15:01:33.622526 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:33.622285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:59165)
0810 15:01:33.622436 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:33.622479 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260810 15:01:33.735302 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:33.735085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:43503)
0810 15:01:33.735220 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:33.735258 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260810 15:01:34.557185 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:34.556913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:50541)
0810 15:01:34.557087 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:34.557134 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260810 15:01:34.623265 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:34.622989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:39395)
0810 15:01:34.623145 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:34.623198 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260810 15:01:34.736040 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:34.735783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:55455)
0810 15:01:34.735936 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:34.735988 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260810 15:01:35.557847 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:35.557608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:37855)
0810 15:01:35.557752 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:35.557798 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260810 15:01:35.623950 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:35.623694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:35755)
0810 15:01:35.623872 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:35.623904 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260810 15:01:35.736850 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:35.736630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:51625)
0810 15:01:35.736759 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:35.736800 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260810 15:01:36.558511 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:36.558234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:45559)
0810 15:01:36.558397 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:36.558438 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260810 15:01:36.624657 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:36.624412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:45055)
0810 15:01:36.624566 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:36.624610 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260810 15:01:36.737496 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:36.737270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:40951)
0810 15:01:36.737412 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:36.737450 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260810 15:01:37.234512 25020 raft_consensus.cc:482] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:37.234619 25020 raft_consensus.cc:504] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f5ffe74d1845f9a1967eb8bb9fc481" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 46119 } } peers { permanent_uuid: "508d86c66f3e43b1a9b8599aeef4575c" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35067 } } peers { permanent_uuid: "380b5ee88abd43808c61196b03fb8b8b" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 43729 } }
I20260810 15:01:37.234799 25020 leader_election.cc:288] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119), 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067)
W20260810 15:01:37.235221 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73f5ffe74d1845f9a1967eb8bb9fc481 (127.28.193.194:46119): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
W20260810 15:01:37.235299 24282 leader_election.cc:341] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 508d86c66f3e43b1a9b8599aeef4575c (127.28.193.195:35067): Not found: Tablet not found: e7ab0462791346cb98602ca9d159ba31
I20260810 15:01:37.235328 24282 leader_election.cc:302] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 380b5ee88abd43808c61196b03fb8b8b; no voters: 508d86c66f3e43b1a9b8599aeef4575c, 73f5ffe74d1845f9a1967eb8bb9fc481
I20260810 15:01:37.235430 25020 raft_consensus.cc:2726] T e7ab0462791346cb98602ca9d159ba31 P 380b5ee88abd43808c61196b03fb8b8b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:01:37.559109 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:37.558911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:49003)
0810 15:01:37.559029 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:37.559063 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260810 15:01:37.625341 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:37.625122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:42673)
0810 15:01:37.625240 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:37.625277 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260810 15:01:37.738142 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:37.737909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:50685)
0810 15:01:37.738059 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:37.738098 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260810 15:01:38.559731 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:38.559531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:47261)
0810 15:01:38.559645 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:38.559683 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260810 15:01:38.626013 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:38.625770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:51631)
0810 15:01:38.625932 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:38.625970 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260810 15:01:38.738808 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:38.738617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:55993)
0810 15:01:38.738735 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:38.738765 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260810 15:01:39.560374 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:39.560136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:59879)
0810 15:01:39.560283 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:39.560321 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260810 15:01:39.626668 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:39.626424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:33743)
0810 15:01:39.626580 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:39.626619 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260810 15:01:39.739475 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:39.739256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:55697)
0810 15:01:39.739396 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:39.739430 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260810 15:01:40.561138 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:40.560893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:32773)
0810 15:01:40.561044 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:40.561087 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260810 15:01:40.627341 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:40.627118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:40173)
0810 15:01:40.627256 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:40.627294 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260810 15:01:40.740151 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:40.739912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:38311)
0810 15:01:40.740061 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:40.740101 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260810 15:01:41.561928 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:41.561664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:46977)
0810 15:01:41.561825 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:41.561866 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260810 15:01:41.627969 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:41.627731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:59843)
0810 15:01:41.627886 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:41.627921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260810 15:01:41.740808 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:41.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:38467)
0810 15:01:41.740720 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:41.740762 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260810 15:01:42.562731 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:42.562522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.193:40695)
0810 15:01:42.562643 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:42.562683 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260810 15:01:42.628686 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:42.628486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.194:46301)
0810 15:01:42.628606 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:42.628641 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260810 15:01:42.734372 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 24208
I20260810 15:01:42.742146 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:42.741961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44961 (local address 127.28.193.195:50169)
0810 15:01:42.742070 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:42.742102 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.254:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260810 15:01:42.745852 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 24498
I20260810 15:01:42.753815 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 24727
2026-08-10T15:01:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58890 ms)
[----------] 30 tests from SecurityITest (233412 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:42 dist-test-slave-cvwc krb5kdc[25056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:42 dist-test-slave-cvwc krb5kdc[25056](info): set up 2 sockets
Aug 10 15:01:42 dist-test-slave-cvwc krb5kdc[25056](info): commencing operation
krb5kdc: starting...
W20260810 15:01:43.942739 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:01:44.058631 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35971
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:35971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:44.138837 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:44.139034 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:44.139053 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:44.140115 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:44.140146 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:44.140158 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:44.140169 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:44.140185 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:44.141572 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:35971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:44.141800 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:44.141984 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:44.146549 25091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.146586 25087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.146540 25088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.146704 25077 server_base.cc:1034] running on GCE node
I20260810 15:01:44.147076 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:44.147337 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:44.148473 25077 hybrid_clock.cc:648] HybridClock initialized: now 1786374104148460 us; error 28 us; skew 500 ppm
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:44.150789 25077 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:01:44.151194 25077 webserver.cc:460] Webserver started at http://127.28.193.254:39657/ using document root <none> and password file <none>
I20260810 15:01:44.151383 25077 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:44.151429 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:44.151551 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:44.152541 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "17a2c4759afc4bc2838f78c1f64291d2"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "5c0b0726473f88decce78d769a228eac"
server_key_iv: "c5c07e168ac652a721478aa3d6565a4c"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.152873 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "17a2c4759afc4bc2838f78c1f64291d2"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "5c0b0726473f88decce78d769a228eac"
server_key_iv: "c5c07e168ac652a721478aa3d6565a4c"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.154141 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:44.154803 25106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.154968 25077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:44.155048 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "17a2c4759afc4bc2838f78c1f64291d2"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "5c0b0726473f88decce78d769a228eac"
server_key_iv: "c5c07e168ac652a721478aa3d6565a4c"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.155108 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:44.173130 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:44.175127 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:44.175273 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:44.179121 25077 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:35971
I20260810 15:01:44.179157 25201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:35971 every 8 connection(s)
I20260810 15:01:44.179490 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:44.180332 25203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:44.182639 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: Bootstrap starting.
I20260810 15:01:44.183210 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 25077
I20260810 15:01:44.183255 25203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:44.183312 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:44.183475 29447 external_mini_cluster.cc:1428] Setting key 76212d0c6d15a2f4e6cda75cb008a486
I20260810 15:01:44.183512 25203 log.cc:824] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:44.184322 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: No bootstrap required, opened a new log
I20260810 15:01:44.186760 25203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } }
I20260810 15:01:44.186930 25203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:44.187014 25203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a2c4759afc4bc2838f78c1f64291d2, State: Initialized, Role: FOLLOWER
I20260810 15:01:44.187145 25203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } }
I20260810 15:01:44.187227 25203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:44.187251 25203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:44.187305 25203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:44.187839 25203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } }
I20260810 15:01:44.187943 25203 leader_election.cc:302] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a2c4759afc4bc2838f78c1f64291d2; no voters:
I20260810 15:01:44.188087 25203 leader_election.cc:288] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:44.188128 25209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:44.188288 25209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [term 1 LEADER]: Becoming Leader. State: Replica: 17a2c4759afc4bc2838f78c1f64291d2, State: Running, Role: LEADER
I20260810 15:01:44.188334 25203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:44.188444 25209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } }
I20260810 15:01:44.188834 25212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a2c4759afc4bc2838f78c1f64291d2. Latest consensus state: current_term: 1 leader_uuid: "17a2c4759afc4bc2838f78c1f64291d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } } }
I20260810 15:01:44.188910 25212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:44.189045 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a2c4759afc4bc2838f78c1f64291d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a2c4759afc4bc2838f78c1f64291d2" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35971 } } }
I20260810 15:01:44.189100 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:44.189692 25219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:44.190201 25219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:44.192278 25219 catalog_manager.cc:1347] Generated new cluster ID: a8383bb63e074c18b777389006443365
I20260810 15:01:44.192332 25219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:44.196656 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.183941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.28.193.254:35971)
0810 15:01:44.184095 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.184104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.184142 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:01:44.184695 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.184705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.184793 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.184947 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:44.185498 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.186890 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.187740 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.188341 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.188772 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.191768 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:44.191773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:44.191773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:44.191794 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:44.194918 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.195236 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.195239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.195240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.195335 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.195506 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.195507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.195507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.195733 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:44.196359 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.196419 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.196518 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260810 15:01:44.224355 25219 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:44.224879 25219 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:44.227422 25219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: Generated new TSK 0
I20260810 15:01:44.227583 25219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:01:44.264829 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:44.369885 25239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:44.370106 25239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:44.370131 25239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:44.371305 25239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:44.371356 25239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:44.371385 25239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:44.372819 25239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:44.373042 25239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:44.373272 25239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:44.378194 25248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.378317 25250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.378274 25254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.378412 25239 server_base.cc:1034] running on GCE node
I20260810 15:01:44.378748 25239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:44.378990 25239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:44.380129 25239 hybrid_clock.cc:648] HybridClock initialized: now 1786374104380115 us; error 27 us; skew 500 ppm
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:44.382506 25239 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:01:44.382908 25239 webserver.cc:460] Webserver started at http://127.28.193.193:41781/ using document root <none> and password file <none>
I20260810 15:01:44.383098 25239 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:44.383143 25239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:44.383268 25239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:44.384269 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "f7a579529ccb400b9644d11016b32ada"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "06f4c61303284bc4c3af19a06bd572e0"
server_key_iv: "7f41dbebeecc8b77b65fbf56ba171e40"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.384600 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "f7a579529ccb400b9644d11016b32ada"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "06f4c61303284bc4c3af19a06bd572e0"
server_key_iv: "7f41dbebeecc8b77b65fbf56ba171e40"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.385859 25239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:44.386574 25266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.386722 25239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:44.386842 25239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "f7a579529ccb400b9644d11016b32ada"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "06f4c61303284bc4c3af19a06bd572e0"
server_key_iv: "7f41dbebeecc8b77b65fbf56ba171e40"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.386912 25239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:44.400336 25239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:44.402320 25239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:44.402444 25239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:44.402637 25239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:44.402912 25239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:44.402944 25239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.402971 25239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:44.402995 25239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.408288 25239 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:40651
I20260810 15:01:44.408349 25458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:40651 every 8 connection(s)
I20260810 15:01:44.408612 25239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:44.410998 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 25239
I20260810 15:01:44.411096 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:44.411266 29447 external_mini_cluster.cc:1428] Setting key 2cdeec39290261eee985338a41ff58ca
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:44.417910 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.409286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:46381 (local address 127.28.193.254:35971)
0810 15:01:44.409340 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.409342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.409528 (+ 186us) server_negotiation.cc:407] Connection header received
0810 15:01:44.409987 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.409988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.409998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.410051 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:44.410954 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.411558 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.412633 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.412898 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.412971 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.415197 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:44.415199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:44.415199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:44.415205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:44.416426 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.416862 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.416863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.416863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.416911 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.417202 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.417203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.417203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.417355 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:44.417425 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.417762 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.417844 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:01:44.418115 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.409254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35971 (local address 127.28.193.193:46381)
0810 15:01:44.409414 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.409453 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.409915 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.410102 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.410105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.410634 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:44.410844 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.410851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.411646 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.411647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.412535 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.412538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.412973 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.412974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.413071 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.413506 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:44.413516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:44.415096 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:44.416502 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.416510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.416512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.416791 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.416970 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.416971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.416971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.417102 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.417430 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:44.417437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:44.417593 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.417706 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.417899 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260810 15:01:44.418643 25462 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35971
I20260810 15:01:44.418743 25462 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:44.418936 25462 heartbeater.cc:511] Master 127.28.193.254:35971 requested a full tablet report, sending...
I20260810 15:01:44.419433 25133 ts_manager.cc:194] Registered new tserver with Master: f7a579529ccb400b9644d11016b32ada (127.28.193.193:40651)
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
I20260810 15:01:44.420032 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:46381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:01:44.434373 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:44.514442 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:44.514638 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:44.514659 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:44.515727 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:44.515759 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:44.515784 25470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:44.517167 25470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:44.517350 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:44.517529 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:44.522187 25481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.522259 25470 server_base.cc:1034] running on GCE node
W20260810 15:01:44.522224 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.522447 25483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.522642 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:44.522890 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:44.524052 25470 hybrid_clock.cc:648] HybridClock initialized: now 1786374104524039 us; error 28 us; skew 500 ppm
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:44.526479 25470 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:01:44.526916 25470 webserver.cc:460] Webserver started at http://127.28.193.194:46641/ using document root <none> and password file <none>
I20260810 15:01:44.527128 25470 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:44.527171 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:44.527302 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:44.528363 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "8f01a7087b5e404aa53b97a8864f47ce"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7796699660e06897a8c88669b6576029"
server_key_iv: "cbcbd5beb90e06805c5c88460ae43cf4"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.528716 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "8f01a7087b5e404aa53b97a8864f47ce"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7796699660e06897a8c88669b6576029"
server_key_iv: "cbcbd5beb90e06805c5c88460ae43cf4"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.530040 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:44.530771 25497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.530942 25470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:44.531023 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "8f01a7087b5e404aa53b97a8864f47ce"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7796699660e06897a8c88669b6576029"
server_key_iv: "cbcbd5beb90e06805c5c88460ae43cf4"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.531081 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:44.552162 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:44.554405 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:44.554562 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:44.554801 25470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:44.555114 25470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:44.555150 25470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.555177 25470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:44.555200 25470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.560926 25470 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:33715
I20260810 15:01:44.560988 25695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:33715 every 8 connection(s)
I20260810 15:01:44.561283 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:44.568993 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 25470
I20260810 15:01:44.569093 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:44.569270 29447 external_mini_cluster.cc:1428] Setting key 5dbc43bc4aca42bd82e2ac439c7d4a03
I20260810 15:01:44.570477 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.561973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:41509 (local address 127.28.193.254:35971)
0810 15:01:44.562053 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.562056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.562230 (+ 174us) server_negotiation.cc:407] Connection header received
0810 15:01:44.562713 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.562714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.562726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.562787 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:44.563719 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.564240 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.565275 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.565475 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.565547 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.567685 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:44.567686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:44.567686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:44.567693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:44.568895 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.569409 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.569411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.569411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.569475 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.569780 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.569782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.569782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.569929 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:44.569993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.570314 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.570393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:01:44.570577 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.561949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35971 (local address 127.28.193.194:41509)
0810 15:01:44.562112 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.562149 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.562616 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.562849 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.562852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.563415 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:44.563614 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.563619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.564353 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.564354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.565168 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.565170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.565579 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.565580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.565697 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.566211 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:44.566220 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:44.567583 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:44.568991 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.568999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.569002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.569297 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.569538 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.569539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.569539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.569676 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.569998 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:44.570006 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:44.570155 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.570263 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.570397 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260810 15:01:44.571169 25699 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35971
I20260810 15:01:44.571271 25699 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:44.571458 25699 heartbeater.cc:511] Master 127.28.193.254:35971 requested a full tablet report, sending...
I20260810 15:01:44.571892 25133 ts_manager.cc:194] Registered new tserver with Master: 8f01a7087b5e404aa53b97a8864f47ce (127.28.193.194:33715)
I20260810 15:01:44.572255 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:41509
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:01:44.594439 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:44.675230 25704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:44.675436 25704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:44.675457 25704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:44.676527 25704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:44.676558 25704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:44.676581 25704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:44.678015 25704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:35971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:44.678211 25704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:44.678391 25704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:44.683387 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.683380 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.683380 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.683651 25704 server_base.cc:1034] running on GCE node
I20260810 15:01:44.683843 25704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:44.684060 25704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:44.685216 25704 hybrid_clock.cc:648] HybridClock initialized: now 1786374104685197 us; error 33 us; skew 500 ppm
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:44.687777 25704 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:01:44.688259 25704 webserver.cc:460] Webserver started at http://127.28.193.195:34471/ using document root <none> and password file <none>
I20260810 15:01:44.688503 25704 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:44.688551 25704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:44.688678 25704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:44.689916 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "06baf0c070ad4c44b8605c3f2a6687ea"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "070fd44433b8ba33349b8ec5581a51e0"
server_key_iv: "da43a7c7b3f2278ece54b9035c7333fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.690260 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "06baf0c070ad4c44b8605c3f2a6687ea"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "070fd44433b8ba33349b8ec5581a51e0"
server_key_iv: "da43a7c7b3f2278ece54b9035c7333fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.691526 25704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:44.692238 25731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.692407 25704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:44.692488 25704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "06baf0c070ad4c44b8605c3f2a6687ea"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "070fd44433b8ba33349b8ec5581a51e0"
server_key_iv: "da43a7c7b3f2278ece54b9035c7333fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.692555 25704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:44.705225 25704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:44.707207 25704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:44.707352 25704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:44.707559 25704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:44.707893 25704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:44.707930 25704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.707958 25704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:44.707975 25704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.713378 25704 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:40519
I20260810 15:01:44.713415 25929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:40519 every 8 connection(s)
I20260810 15:01:44.713738 25704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:44.719373 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 25704
I20260810 15:01:44.719472 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:44.719636 29447 external_mini_cluster.cc:1428] Setting key 2d25fe6e199290191eb1a4ef72307bca
Aug 10 15:01:44 dist-test-slave-cvwc krb5kdc[25056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374104, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:44.722865 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.714456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:48077 (local address 127.28.193.254:35971)
0810 15:01:44.714544 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.714546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.714688 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:01:44.715352 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.715354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.715367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.715422 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:44.716374 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.716895 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.717878 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.718073 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.718120 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.720279 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:44.720281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:44.720281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:44.720289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:44.721441 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.721885 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.721886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.721886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.721949 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.722189 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.722190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.722191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.722313 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:44.722391 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.722671 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.722772 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 15:01:44.723021 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.714393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:35971 (local address 127.28.193.195:48077)
0810 15:01:44.714556 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.714599 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.715215 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.715491 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.715495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.716073 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:44.716267 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.716272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.716961 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.716961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.717796 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.717799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.718134 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.718135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.718239 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.718687 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:44.718696 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:44.720149 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:44.721520 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.721527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.721530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.721813 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.722010 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:44.722012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:44.722012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:44.722126 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:44.722377 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:44.722384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:44.722528 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.722644 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.722799 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260810 15:01:44.723443 25932 heartbeater.cc:347] Connected to a master server at 127.28.193.254:35971
I20260810 15:01:44.723537 25932 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:44.723709 25932 heartbeater.cc:511] Master 127.28.193.254:35971 requested a full tablet report, sending...
I20260810 15:01:44.724053 25133 ts_manager.cc:194] Registered new tserver with Master: 06baf0c070ad4c44b8605c3f2a6687ea (127.28.193.195:40519)
I20260810 15:01:44.724403 25133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:48077
I20260810 15:01:44.731087 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:44.738006 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.732299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.28.193.254:35971)
0810 15:01:44.732361 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.732363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.732501 (+ 138us) server_negotiation.cc:407] Connection header received
0810 15:01:44.732557 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.732559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.732570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.732631 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:44.733270 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.733809 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.734498 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.734705 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.734802 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.735872 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:44.735874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:44.735874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:44.735880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:44.737092 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.737382 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.737382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.737383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.737428 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:44.737545 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:44.737545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:44.737545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:44.737666 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:44.737781 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.737878 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.737937 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:01:44.739313 25133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:44.740832 25133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:44.747109 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:40519 (local address 127.0.0.1:60754)
0810 15:01:44.743600 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.743610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.743677 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.743855 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.743856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.743858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:44.743927 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.743929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.744863 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.744864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.746015 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.746018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.746820 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.746821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.746899 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.746913 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.746980 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.747032 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":111,"threads_started":1}
I20260810 15:01:44.747133 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.28.193.195:40519)
0810 15:01:44.743574 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.743578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.743679 (+ 101us) server_negotiation.cc:407] Connection header received
0810 15:01:44.743758 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.743760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.743776 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.743858 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:44.744007 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.744775 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.746086 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.746742 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.746814 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.746871 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.747013 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.747055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260810 15:01:44.747545 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:33715 (local address 127.0.0.1:55594)
0810 15:01:44.743823 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.743833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.743878 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.744140 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.744141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.744143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:44.744186 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.744187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.745412 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.745413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.746420 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.746423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.747326 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.747327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.747387 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.747390 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.747434 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.747470 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":81,"threads_started":1}
I20260810 15:01:44.747648 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.28.193.193:40651)
0810 15:01:44.743522 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.743528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.743539 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:01:44.743605 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.743608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.743627 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.743704 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:44.744110 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.745181 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.746364 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.747025 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.747096 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.747144 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.747509 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.747557 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260810 15:01:44.747927 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.28.193.194:33715)
0810 15:01:44.743556 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.743559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.743933 (+ 374us) server_negotiation.cc:407] Connection header received
0810 15:01:44.743980 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.743982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.743998 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.744061 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:44.744273 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.745317 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.746506 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.747192 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.747249 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.747296 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.747811 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.747852 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260810 15:01:44.748040 25813 tablet_service.cc:1467] Processing CreateTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f (DEFAULT_TABLE table=test-table [id=acb36d07031d4de982848e72a55beec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:44.748409 25813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271f9e97a085494fb7d4f5dd1f73d08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:44.750136 25569 tablet_service.cc:1467] Processing CreateTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f (DEFAULT_TABLE table=test-table [id=acb36d07031d4de982848e72a55beec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:44.750450 25569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271f9e97a085494fb7d4f5dd1f73d08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:44.750962 25965 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Bootstrap starting.
I20260810 15:01:44.751579 25965 tablet_bootstrap.cc:654] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:44.752068 25965 log.cc:824] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:44.752694 25965 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: No bootstrap required, opened a new log
I20260810 15:01:44.752748 25965 ts_tablet_manager.cc:1397] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:44.753189 25969 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Bootstrap starting.
I20260810 15:01:44.753257 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.743150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:40651 (local address 127.0.0.1:40534)
0810 15:01:44.743291 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.743335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.743404 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.743754 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.743758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.743760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:44.743967 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.743969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.745291 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.745292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.746288 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.746291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.747182 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.747183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.747431 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.747434 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.747483 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.747514 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
I20260810 15:01:44.753968 25345 tablet_service.cc:1467] Processing CreateTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f (DEFAULT_TABLE table=test-table [id=acb36d07031d4de982848e72a55beec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:44.754139 25969 tablet_bootstrap.cc:654] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:44.754267 25345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 271f9e97a085494fb7d4f5dd1f73d08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:44.754714 25969 log.cc:824] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:44.754592 25965 raft_consensus.cc:348] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.754776 25965 raft_consensus.cc:374] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:44.754810 25965 raft_consensus.cc:729] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06baf0c070ad4c44b8605c3f2a6687ea, State: Initialized, Role: FOLLOWER
I20260810 15:01:44.754948 25965 consensus_queue.cc:260] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.755219 25965 ts_tablet_manager.cc:1428] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:01:44.755378 25932 heartbeater.cc:503] Master 127.28.193.254:35971 was elected leader, sending a full tablet report...
I20260810 15:01:44.755478 25969 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: No bootstrap required, opened a new log
I20260810 15:01:44.755524 25969 ts_tablet_manager.cc:1397] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:01:44.756953 25969 raft_consensus.cc:348] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.757083 25969 raft_consensus.cc:374] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:44.757103 25969 raft_consensus.cc:729] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f01a7087b5e404aa53b97a8864f47ce, State: Initialized, Role: FOLLOWER
I20260810 15:01:44.757220 25969 consensus_queue.cc:260] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.757308 25974 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: Bootstrap starting.
I20260810 15:01:44.757509 25969 ts_tablet_manager.cc:1428] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:01:44.757537 25699 heartbeater.cc:503] Master 127.28.193.254:35971 was elected leader, sending a full tablet report...
I20260810 15:01:44.757969 25974 tablet_bootstrap.cc:654] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:44.758451 25974 log.cc:824] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:44.759181 25974 tablet_bootstrap.cc:492] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: No bootstrap required, opened a new log
I20260810 15:01:44.759232 25974 ts_tablet_manager.cc:1397] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:44.760461 25974 raft_consensus.cc:348] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.760584 25974 raft_consensus.cc:374] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:44.760604 25974 raft_consensus.cc:729] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a579529ccb400b9644d11016b32ada, State: Initialized, Role: FOLLOWER
I20260810 15:01:44.760694 25974 consensus_queue.cc:260] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.760931 25974 ts_tablet_manager.cc:1428] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:44.760995 25462 heartbeater.cc:503] Master 127.28.193.254:35971 was elected leader, sending a full tablet report...
I20260810 15:01:44.781100 25984 raft_consensus.cc:482] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:44.781225 25984 raft_consensus.cc:504] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.781556 25984 leader_election.cc:288] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06baf0c070ad4c44b8605c3f2a6687ea (127.28.193.195:40519), 8f01a7087b5e404aa53b97a8864f47ce (127.28.193.194:33715)
I20260810 15:01:44.784227 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.781810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:43639 (local address 127.28.193.195:40519)
0810 15:01:44.781851 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.781853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.781915 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:01:44.781971 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.781972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.781981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.782029 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:44.782158 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.782618 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.783545 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.783922 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.783981 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.783990 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.784130 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.784163 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:01:44.784235 25466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.781738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:40519 (local address 127.28.193.193:43639)
0810 15:01:44.781857 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.781868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.781921 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.782035 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.782036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.782037 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:44.782105 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.782107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.782687 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.782688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.783461 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.783463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.783994 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.783996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.784073 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.784076 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.784119 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.784159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":5}
I20260810 15:01:44.784478 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.781818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:34257 (local address 127.28.193.194:33715)
0810 15:01:44.781890 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.781892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.782048 (+ 156us) server_negotiation.cc:407] Connection header received
0810 15:01:44.782119 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.782120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.782131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.782185 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:44.782337 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.782802 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.783712 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.784085 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.784155 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.784164 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.784353 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.784402 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:01:44.784544 25857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271f9e97a085494fb7d4f5dd1f73d08f" candidate_uuid: "f7a579529ccb400b9644d11016b32ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" is_pre_election: true
I20260810 15:01:44.784696 25857 raft_consensus.cc:2454] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a579529ccb400b9644d11016b32ada in term 0.
I20260810 15:01:44.784698 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.781781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:33715 (local address 127.28.193.193:34257)
0810 15:01:44.781971 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.781981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:44.782044 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:44.782195 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:44.782197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:44.782198 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:44.782270 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.782272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.782892 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.782893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.783618 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:44.783621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.784169 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.784170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:44.784247 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.784256 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 15:01:44.784571 (+ 315us) client_negotiation.cc:240] Negotiation successful
0810 15:01:44.784615 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":124,"threads_started":1}
I20260810 15:01:44.784932 25272 leader_election.cc:302] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baf0c070ad4c44b8605c3f2a6687ea, f7a579529ccb400b9644d11016b32ada; no voters:
I20260810 15:01:44.785058 25984 raft_consensus.cc:2781] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:44.785112 25984 raft_consensus.cc:482] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:44.785043 25604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271f9e97a085494fb7d4f5dd1f73d08f" candidate_uuid: "f7a579529ccb400b9644d11016b32ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f01a7087b5e404aa53b97a8864f47ce" is_pre_election: true
I20260810 15:01:44.785127 25984 raft_consensus.cc:3037] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:44.785187 25604 raft_consensus.cc:2454] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7a579529ccb400b9644d11016b32ada in term 0.
I20260810 15:01:44.785867 25984 raft_consensus.cc:504] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.785992 25984 leader_election.cc:288] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [CANDIDATE]: Term 1 election: Requested vote from peers 06baf0c070ad4c44b8605c3f2a6687ea (127.28.193.195:40519), 8f01a7087b5e404aa53b97a8864f47ce (127.28.193.194:33715)
I20260810 15:01:44.786124 25857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271f9e97a085494fb7d4f5dd1f73d08f" candidate_uuid: "f7a579529ccb400b9644d11016b32ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baf0c070ad4c44b8605c3f2a6687ea"
I20260810 15:01:44.786193 25857 raft_consensus.cc:3037] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:44.786180 25604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "271f9e97a085494fb7d4f5dd1f73d08f" candidate_uuid: "f7a579529ccb400b9644d11016b32ada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f01a7087b5e404aa53b97a8864f47ce"
I20260810 15:01:44.786296 25604 raft_consensus.cc:3037] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:44.786984 25857 raft_consensus.cc:2454] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a579529ccb400b9644d11016b32ada in term 1.
I20260810 15:01:44.787170 25272 leader_election.cc:302] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baf0c070ad4c44b8605c3f2a6687ea, f7a579529ccb400b9644d11016b32ada; no voters:
I20260810 15:01:44.787246 25604 raft_consensus.cc:2454] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7a579529ccb400b9644d11016b32ada in term 1.
I20260810 15:01:44.787276 25984 raft_consensus.cc:2781] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:44.787463 25984 raft_consensus.cc:686] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [term 1 LEADER]: Becoming Leader. State: Replica: f7a579529ccb400b9644d11016b32ada, State: Running, Role: LEADER
I20260810 15:01:44.787580 25984 consensus_queue.cc:237] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } }
I20260810 15:01:44.788537 25133 catalog_manager.cc:5495] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada reported cstate change: term changed from 0 to 1, leader changed from <none> to f7a579529ccb400b9644d11016b32ada (127.28.193.193). New cstate: current_term: 1 leader_uuid: "f7a579529ccb400b9644d11016b32ada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7a579529ccb400b9644d11016b32ada" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 40651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:44.804237 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.802052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.28.193.193:40651)
0810 15:01:44.802099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:44.802101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:44.802110 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:01:44.802173 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:44.802174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:44.802192 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:44.802254 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:44.802374 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.802840 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.803555 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:44.803757 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:44.803810 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:44.803959 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:44.804053 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:44.804120 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:44.804147 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 15:01:44.804166 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:01:44.806402 25857 raft_consensus.cc:1261] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 1 FOLLOWER]: Refusing update from remote peer f7a579529ccb400b9644d11016b32ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:44.806427 25604 raft_consensus.cc:1261] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 1 FOLLOWER]: Refusing update from remote peer f7a579529ccb400b9644d11016b32ada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:44.806740 25986 consensus_queue.cc:1035] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "06baf0c070ad4c44b8605c3f2a6687ea" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:44.806864 25984 consensus_queue.cc:1035] T 271f9e97a085494fb7d4f5dd1f73d08f P f7a579529ccb400b9644d11016b32ada [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f01a7087b5e404aa53b97a8864f47ce" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:44.809451 25998 mvcc.cc:205] Tried to move back new op lower bound from 7316988333284487168 to 7316988333210513408. Current Snapshot: MvccSnapshot[applied={T|T < 7316988333284487168}]
I20260810 15:01:44.811460 25132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:44.811558 25132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:44.812268 25132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17a2c4759afc4bc2838f78c1f64291d2: Sending DeleteTablet for 3 replicas of tablet 271f9e97a085494fb7d4f5dd1f73d08f
I20260810 15:01:44.812567 25345 tablet_service.cc:1514] Processing DeleteTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:44 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:40534
I20260810 15:01:44.812585 25569 tablet_service.cc:1514] Processing DeleteTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:44 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:55594
I20260810 15:01:44.812894 25813 tablet_service.cc:1514] Processing DeleteTablet for tablet 271f9e97a085494fb7d4f5dd1f73d08f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:44 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:60754
I20260810 15:01:44.813100 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 25239
I20260810 15:01:44.824265 26023 tablet_replica.cc:330] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: stopping tablet replica
I20260810 15:01:44.824463 26023 raft_consensus.cc:2229] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:44.824841 26023 raft_consensus.cc:2258] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:44.826181 26022 tablet_replica.cc:330] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: stopping tablet replica
I20260810 15:01:44.826305 26022 raft_consensus.cc:2229] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:44.826423 26022 raft_consensus.cc:2258] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:44.826529 26023 ts_tablet_manager.cc:1905] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:44.826931 26022 ts_tablet_manager.cc:1905] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:44.828163 26023 ts_tablet_manager.cc:1918] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:44.828230 26023 log.cc:1196] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/271f9e97a085494fb7d4f5dd1f73d08f
I20260810 15:01:44.828470 26023 ts_tablet_manager.cc:1939] T 271f9e97a085494fb7d4f5dd1f73d08f P 06baf0c070ad4c44b8605c3f2a6687ea: Deleting consensus metadata
I20260810 15:01:44.828480 26022 ts_tablet_manager.cc:1918] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:44.828547 26022 log.cc:1196] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/271f9e97a085494fb7d4f5dd1f73d08f
I20260810 15:01:44.828792 26022 ts_tablet_manager.cc:1939] T 271f9e97a085494fb7d4f5dd1f73d08f P 8f01a7087b5e404aa53b97a8864f47ce: Deleting consensus metadata
I20260810 15:01:44.828887 25115 catalog_manager.cc:4841] TS 06baf0c070ad4c44b8605c3f2a6687ea (127.28.193.195:40519): tablet 271f9e97a085494fb7d4f5dd1f73d08f (table test-table [id=acb36d07031d4de982848e72a55beec1]) successfully deleted
I20260810 15:01:44.829195 25112 catalog_manager.cc:4841] TS 8f01a7087b5e404aa53b97a8864f47ce (127.28.193.194:33715): tablet 271f9e97a085494fb7d4f5dd1f73d08f (table test-table [id=acb36d07031d4de982848e72a55beec1]) successfully deleted
W20260810 15:01:44.831331 25116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:40651 (error 108)
I20260810 15:01:44.831597 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 25470
I20260810 15:01:44.831821 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:44.831704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:40651 (local address 127.0.0.1:40552)
0810 15:01:44.831754 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:44.831790 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:01:44.831915 25116 catalog_manager.cc:4584] TS f7a579529ccb400b9644d11016b32ada (127.28.193.193:40651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 271f9e97a085494fb7d4f5dd1f73d08f: Network error: Client connection negotiation failed: client connection to 127.28.193.193:40651: connect: Connection refused (error 111)
I20260810 15:01:44.844705 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 25704
I20260810 15:01:44.866914 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 25077
2026-08-10T15:01:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2114 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T15:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:44Z Disabled control of system clock
I20260810 15:01:44.899173 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33301
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:33301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:44.979104 26031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:44.979327 26031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:44.979346 26031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:44.980427 26031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:44.980459 26031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:44.980469 26031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:44.980480 26031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:44.980490 26031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:44.981793 26031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:33301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:44.981993 26031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:44.982223 26031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:44.986976 26044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:44.987028 26039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.987186 26031 server_base.cc:1034] running on GCE node
W20260810 15:01:44.987241 26040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:44.987506 26031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:44.987777 26031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:44.988927 26031 hybrid_clock.cc:648] HybridClock initialized: now 1786374104988914 us; error 29 us; skew 500 ppm
I20260810 15:01:44.990187 26031 webserver.cc:460] Webserver started at http://127.28.193.254:40951/ using document root <none> and password file <none>
I20260810 15:01:44.990413 26031 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:44.990463 26031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:44.990582 26031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:44.991677 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "4c77f6e484b44559af77b07531434c0b"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7a9a53c2958bd79dff91fefcc6f6a69b"
server_key_iv: "21500252b8b075522aee25a8596ed367"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.992051 26031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "4c77f6e484b44559af77b07531434c0b"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7a9a53c2958bd79dff91fefcc6f6a69b"
server_key_iv: "21500252b8b075522aee25a8596ed367"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.993386 26031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:44.994202 26059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.994364 26031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:44.994455 26031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "4c77f6e484b44559af77b07531434c0b"
format_stamp: "Formatted at 2026-08-10 15:01:44 on dist-test-slave-cvwc"
server_key: "7a9a53c2958bd79dff91fefcc6f6a69b"
server_key_iv: "21500252b8b075522aee25a8596ed367"
server_key_version: "encryptionkey@0"
I20260810 15:01:44.994530 26031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:45.036036 26031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:45.036375 26031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:45.036509 26031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:45.040370 26031 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:33301
I20260810 15:01:45.040412 26161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:33301 every 8 connection(s)
I20260810 15:01:45.040736 26031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:45.041788 26162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:45.043712 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 26031
I20260810 15:01:45.043833 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:45.044025 29447 external_mini_cluster.cc:1428] Setting key 50b079e8bfa1fdb7d5bbd4d6ecdc8cb1
I20260810 15:01:45.044262 26162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: Bootstrap starting.
I20260810 15:01:45.044968 26162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:45.045305 26162 log.cc:824] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:45.046110 26162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: No bootstrap required, opened a new log
I20260810 15:01:45.048290 26162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } }
I20260810 15:01:45.048458 26162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:45.048491 26162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c77f6e484b44559af77b07531434c0b, State: Initialized, Role: FOLLOWER
I20260810 15:01:45.048619 26162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } }
I20260810 15:01:45.048699 26162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:45.048781 26162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:45.048832 26162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:45.049497 26162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } }
I20260810 15:01:45.049608 26162 leader_election.cc:302] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c77f6e484b44559af77b07531434c0b; no voters:
I20260810 15:01:45.049788 26162 leader_election.cc:288] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:45.049973 26175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:45.050073 26162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:45.050163 26175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [term 1 LEADER]: Becoming Leader. State: Replica: 4c77f6e484b44559af77b07531434c0b, State: Running, Role: LEADER
I20260810 15:01:45.050334 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.044533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.28.193.254:33301)
0810 15:01:45.044763 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.044772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.044805 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:01:45.045363 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.045372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.045454 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.045596 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:45.045875 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.047212 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.048047 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.048636 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.048765 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.049431 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:45.049435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:45.049435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:45.049647 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:45.049689 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:45.049704 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:45.049797 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.049889 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.050070 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260810 15:01:45.050335 26175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } }
I20260810 15:01:45.050710 26175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c77f6e484b44559af77b07531434c0b. Latest consensus state: current_term: 1 leader_uuid: "4c77f6e484b44559af77b07531434c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } } }
I20260810 15:01:45.050719 26174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c77f6e484b44559af77b07531434c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c77f6e484b44559af77b07531434c0b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33301 } } }
I20260810 15:01:45.050884 26175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:45.050891 26174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b [sys.catalog]: This master's current role is: LEADER
W20260810 15:01:45.051887 26199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:01:45.051929 26199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:01:45.051990 26186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:45.052170 26186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:45.053560 26186 catalog_manager.cc:1347] Generated new cluster ID: 9d5b70cb6f8a47e8afd7f7eedf6a8750
I20260810 15:01:45.053611 26186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:45.080920 26186 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:45.081453 26186 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:45.088654 26186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: Generated new TSK 0
I20260810 15:01:45.088826 26186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:45.092311 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--builtin_ntp_servers=127.28.193.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:45.172799 26204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:45.172992 26204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:45.173012 26204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:45.174141 26204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:45.174180 26204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:45.174203 26204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:45.175524 26204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:45.175751 26204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:45.176031 26204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:45.180912 26217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:45.180918 26216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.181022 26204 server_base.cc:1034] running on GCE node
W20260810 15:01:45.180924 26221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.181293 26204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:45.181541 26204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:45.182679 26204 hybrid_clock.cc:648] HybridClock initialized: now 1786374105182665 us; error 27 us; skew 500 ppm
I20260810 15:01:45.183895 26204 webserver.cc:460] Webserver started at http://127.28.193.193:40135/ using document root <none> and password file <none>
I20260810 15:01:45.184120 26204 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:45.184170 26204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:45.184302 26204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:45.185315 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "0e12ad8453704daca7b51365d9394795"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "3011aaf7ff5db4d8328b051b0f6560d8"
server_key_iv: "284681ca4703eed098b536d5fba2782a"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.185642 26204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "0e12ad8453704daca7b51365d9394795"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "3011aaf7ff5db4d8328b051b0f6560d8"
server_key_iv: "284681ca4703eed098b536d5fba2782a"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.186901 26204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:45.187563 26231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.187703 26204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:45.187781 26204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "0e12ad8453704daca7b51365d9394795"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "3011aaf7ff5db4d8328b051b0f6560d8"
server_key_iv: "284681ca4703eed098b536d5fba2782a"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.187850 26204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:45.200457 26204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:45.200672 26204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:45.200763 26204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:45.200934 26204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:45.201180 26204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:45.201207 26204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.201233 26204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:45.201249 26204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.206624 26204 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:46343
I20260810 15:01:45.206738 26422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:46343 every 8 connection(s)
I20260810 15:01:45.206982 26204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:45.207888 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 26204
I20260810 15:01:45.207989 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:45.208197 29447 external_mini_cluster.cc:1428] Setting key 1a3b80ddd5779ef218a12f31254f4af2
I20260810 15:01:45.209195 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--builtin_ntp_servers=127.28.193.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:01:45.212580 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.207724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:57633 (local address 127.28.193.254:33301)
0810 15:01:45.207795 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.207797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.208012 (+ 215us) server_negotiation.cc:407] Connection header received
0810 15:01:45.208637 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.208638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.208649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.208710 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:45.209556 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.210090 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.211267 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.211546 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.211611 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.212343 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:45.212344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:45.212345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:45.212350 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:45.212367 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:45.212371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:45.212428 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.212482 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.212505 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:01:45.212739 26425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.207683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33301 (local address 127.28.193.193:57633)
0810 15:01:45.207884 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.207929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.208544 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.208771 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.208775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.208786 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:45.209460 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.209466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.210172 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.210173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.211185 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.211188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.211623 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.211624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.211723 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.212214 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:45.212215 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:45.212242 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:45.212264 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:45.212429 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:45.212431 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:45.212434 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.212473 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.212547 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260810 15:01:45.213217 26423 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33301
I20260810 15:01:45.213299 26423 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:45.213466 26423 heartbeater.cc:511] Master 127.28.193.254:33301 requested a full tablet report, sending...
I20260810 15:01:45.213981 26093 ts_manager.cc:194] Registered new tserver with Master: 0e12ad8453704daca7b51365d9394795 (127.28.193.193:46343)
I20260810 15:01:45.214540 26093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.193:57633
W20260810 15:01:45.290328 26429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:45.290518 26429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:45.290546 26429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:45.291700 26429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:45.291738 26429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:45.291770 26429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:45.293100 26429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:45.293326 26429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:45.293578 26429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:45.298337 26441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:45.298352 26440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.298458 26429 server_base.cc:1034] running on GCE node
W20260810 15:01:45.298355 26444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.298767 26429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:45.298982 26429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:45.300122 26429 hybrid_clock.cc:648] HybridClock initialized: now 1786374105300110 us; error 30 us; skew 500 ppm
I20260810 15:01:45.308254 26429 webserver.cc:460] Webserver started at http://127.28.193.194:38709/ using document root <none> and password file <none>
I20260810 15:01:45.308454 26429 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:45.308494 26429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:45.308589 26429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:45.309571 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "7bf8469ca73f4c1c91f839d2d2c6e782"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "707baa88a8aedb441e46391b555319bb"
server_key_iv: "804ae5831c4d342234a7f323dfb040f8"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.309903 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "7bf8469ca73f4c1c91f839d2d2c6e782"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "707baa88a8aedb441e46391b555319bb"
server_key_iv: "804ae5831c4d342234a7f323dfb040f8"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.311061 26429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:45.311681 26456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.311838 26429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:45.311918 26429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "7bf8469ca73f4c1c91f839d2d2c6e782"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "707baa88a8aedb441e46391b555319bb"
server_key_iv: "804ae5831c4d342234a7f323dfb040f8"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.311972 26429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:45.326599 26429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:45.326840 26429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:45.326938 26429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:45.327109 26429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:45.327366 26429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:45.327395 26429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.327420 26429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:45.327437 26429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.332507 26429 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:42765
I20260810 15:01:45.332574 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:42765 every 8 connection(s)
I20260810 15:01:45.332846 26429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:01:45.335160 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 26429
I20260810 15:01:45.335263 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:45.335480 29447 external_mini_cluster.cc:1428] Setting key 5a5180a28284f16e346c13317f793391
I20260810 15:01:45.336447 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--builtin_ntp_servers=127.28.193.212:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:01:45.338143 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.333511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:36325 (local address 127.28.193.254:33301)
0810 15:01:45.333555 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.333557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.333791 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:01:45.334251 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.334252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.334261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.334316 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:45.334901 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.335438 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.336729 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.336934 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.336983 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.337835 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:45.337837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:45.337837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:45.337843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:45.337865 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:45.337869 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:45.337912 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.338028 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.338060 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:01:45.338495 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.333487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33301 (local address 127.28.193.194:36325)
0810 15:01:45.333654 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.333694 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.334174 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.334368 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.334371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.334378 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:45.334839 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.334844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.335528 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.335529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.336639 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.336643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.337007 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.337008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.337150 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.337624 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:45.337625 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:45.337662 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:45.337687 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:45.337923 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:45.337930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:45.337933 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.338237 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.338335 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260810 15:01:45.338936 26643 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33301
I20260810 15:01:45.339036 26643 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:45.339223 26643 heartbeater.cc:511] Master 127.28.193.254:33301 requested a full tablet report, sending...
I20260810 15:01:45.339605 26093 ts_manager.cc:194] Registered new tserver with Master: 7bf8469ca73f4c1c91f839d2d2c6e782 (127.28.193.194:42765)
I20260810 15:01:45.339956 26093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.194:36325
W20260810 15:01:45.417618 26653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:45.417855 26653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:45.417886 26653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:45.419014 26653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:45.419051 26653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:45.419082 26653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:45.420396 26653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:33301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:45.420627 26653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:45.420892 26653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:45.425586 26664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.425693 26653 server_base.cc:1034] running on GCE node
W20260810 15:01:45.425586 26662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:45.425609 26660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:45.426021 26653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:45.426247 26653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:45.427418 26653 hybrid_clock.cc:648] HybridClock initialized: now 1786374105427402 us; error 26 us; skew 500 ppm
I20260810 15:01:45.428642 26653 webserver.cc:460] Webserver started at http://127.28.193.195:40223/ using document root <none> and password file <none>
I20260810 15:01:45.428876 26653 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:45.428926 26653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:45.429051 26653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:45.430124 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "d63c796ad4de4f11b70abc5f7bc377d0"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "146bf743a655a0837dd312a843afe130"
server_key_iv: "47feebcaa0ef7492ab13de6dc0064b43"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.430455 26653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "d63c796ad4de4f11b70abc5f7bc377d0"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "146bf743a655a0837dd312a843afe130"
server_key_iv: "47feebcaa0ef7492ab13de6dc0064b43"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.431639 26653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:01:45.432294 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.432435 26653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:45.432514 26653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "d63c796ad4de4f11b70abc5f7bc377d0"
format_stamp: "Formatted at 2026-08-10 15:01:45 on dist-test-slave-cvwc"
server_key: "146bf743a655a0837dd312a843afe130"
server_key_iv: "47feebcaa0ef7492ab13de6dc0064b43"
server_key_version: "encryptionkey@0"
I20260810 15:01:45.432577 26653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:45.443102 26653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:45.443351 26653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:45.443467 26653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:45.443657 26653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:45.443976 26653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:45.444015 26653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.444042 26653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:45.444065 26653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:45.449442 26653 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:46435
I20260810 15:01:45.449477 26867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:46435 every 8 connection(s)
I20260810 15:01:45.449815 26653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:45.452162 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 26653
I20260810 15:01:45.452252 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:45.452435 29447 external_mini_cluster.cc:1428] Setting key 3e41dd698c7f8aa957f938826985cb1a
I20260810 15:01:45.454877 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.450521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:50179 (local address 127.28.193.254:33301)
0810 15:01:45.450582 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.450584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.450804 (+ 220us) server_negotiation.cc:407] Connection header received
0810 15:01:45.451343 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.451345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.451358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.451401 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:45.452061 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.452632 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.453590 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.453850 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.453902 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.454609 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:45.454610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:45.454611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:45.454615 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:45.454631 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:45.454635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:45.454682 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.454762 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.454791 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:01:45.455055 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.450507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:33301 (local address 127.28.193.195:50179)
0810 15:01:45.450690 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.450722 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.451236 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.451480 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.451485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.451495 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:45.451968 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.451972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.452713 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.452715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.453502 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.453504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.453928 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.453929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.454061 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.454481 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:45.454481 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:45.454506 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:45.454527 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:45.454688 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:45.454692 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:45.454696 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.454759 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.454858 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260810 15:01:45.455443 26868 heartbeater.cc:347] Connected to a master server at 127.28.193.254:33301
I20260810 15:01:45.455533 26868 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:45.455700 26868 heartbeater.cc:511] Master 127.28.193.254:33301 requested a full tablet report, sending...
I20260810 15:01:45.456027 26093 ts_manager.cc:194] Registered new tserver with Master: d63c796ad4de4f11b70abc5f7bc377d0 (127.28.193.195:46435)
I20260810 15:01:45.456355 26093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.195:50179
I20260810 15:01:45.463991 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:45.468003 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.465284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.28.193.254:33301)
0810 15:01:45.465347 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.465349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.465446 (+ 97us) server_negotiation.cc:407] Connection header received
0810 15:01:45.465489 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.465489 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.465498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.465541 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:45.465699 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.466221 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.466960 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.467141 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.467194 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.467776 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:45.467781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:45.467781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:45.467785 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:45.467798 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:45.467802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:45.467854 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.467912 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.467930 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:01:45.469523 26093 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:45.471050 26093 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:45.476797 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:46435 (local address 127.0.0.1:46056)
0810 15:01:45.473183 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.473227 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.473304 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.473731 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.473732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.473735 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:45.473786 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.473787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.475135 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.475136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.475873 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.475875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.476585 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.476586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.476639 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.476649 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.476696 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.476729 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260810 15:01:45.476873 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.28.193.195:46435)
0810 15:01:45.473560 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.473564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.473574 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:45.473610 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.473612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.473626 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.473875 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:45.473937 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.474975 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.475953 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.476512 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.476582 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.476662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.476713 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.476756 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":112,"threads_started":1}
I20260810 15:01:45.477499 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42765 (local address 127.0.0.1:57474)
0810 15:01:45.473353 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.473363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.473404 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.473612 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.473615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.473617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:45.473701 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.473703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.475015 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.475015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.476154 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.476156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.477272 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.477273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.477334 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.477338 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.477390 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.477432 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":6,"thread_start_us":73,"threads_started":1}
I20260810 15:01:45.477538 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.28.193.194:42765)
0810 15:01:45.473395 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.473400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.473412 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:01:45.473480 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.473483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.473500 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.473592 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:45.473805 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.474926 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.476251 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.476963 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.477033 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.477114 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.477422 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.477460 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260810 15:01:45.477694 26768 tablet_service.cc:1467] Processing CreateTablet for tablet c4824cc12a624e82961ac45cff165fbc (DEFAULT_TABLE table=test-table [id=8614cfc6bc8e4f948ab418516f4c05a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:45.478108 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4824cc12a624e82961ac45cff165fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:45.478384 26540 tablet_service.cc:1467] Processing CreateTablet for tablet c4824cc12a624e82961ac45cff165fbc (DEFAULT_TABLE table=test-table [id=8614cfc6bc8e4f948ab418516f4c05a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:45.478741 26540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4824cc12a624e82961ac45cff165fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:45.481496 26897 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Bootstrap starting.
I20260810 15:01:45.481942 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:46343 (local address 127.0.0.1:33634)
0810 15:01:45.473401 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.473408 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.473441 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.474056 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.474057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.474059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:45.474088 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.474089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.474899 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.474900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.476154 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.476156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.481643 (+ 5487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.481645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.481738 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.481743 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.481813 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.481862 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":4,"thread_start_us":64,"threads_started":1}
I20260810 15:01:45.481985 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.473628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.28.193.193:46343)
0810 15:01:45.473856 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.473860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.473870 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:45.473908 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.473910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.473925 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.473993 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:45.474152 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.474816 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.477153 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.477913 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.477977 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.478056 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.481844 (+ 3788us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.481898 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260810 15:01:45.482157 26898 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Bootstrap starting.
I20260810 15:01:45.482573 26897 tablet_bootstrap.cc:654] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:45.482681 26311 tablet_service.cc:1467] Processing CreateTablet for tablet c4824cc12a624e82961ac45cff165fbc (DEFAULT_TABLE table=test-table [id=8614cfc6bc8e4f948ab418516f4c05a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:45.482910 26897 log.cc:824] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:45.483039 26311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4824cc12a624e82961ac45cff165fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:45.483626 26897 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: No bootstrap required, opened a new log
I20260810 15:01:45.483698 26897 ts_tablet_manager.cc:1397] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:45.486291 26900 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: Bootstrap starting.
I20260810 15:01:45.486222 26897 raft_consensus.cc:348] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.486383 26897 raft_consensus.cc:374] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:45.486423 26897 raft_consensus.cc:729] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d63c796ad4de4f11b70abc5f7bc377d0, State: Initialized, Role: FOLLOWER
I20260810 15:01:45.486554 26897 consensus_queue.cc:260] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.486801 26897 ts_tablet_manager.cc:1428] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:01:45.486963 26868 heartbeater.cc:503] Master 127.28.193.254:33301 was elected leader, sending a full tablet report...
I20260810 15:01:45.487030 26898 tablet_bootstrap.cc:654] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:45.487038 26900 tablet_bootstrap.cc:654] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:45.487326 26898 log.cc:824] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:45.487372 26900 log.cc:824] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:45.488145 26900 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: No bootstrap required, opened a new log
I20260810 15:01:45.488145 26898 tablet_bootstrap.cc:492] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: No bootstrap required, opened a new log
I20260810 15:01:45.488214 26898 ts_tablet_manager.cc:1397] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 15:01:45.488291 26900 ts_tablet_manager.cc:1397] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:45.490679 26898 raft_consensus.cc:348] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.490684 26900 raft_consensus.cc:348] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.490844 26900 raft_consensus.cc:374] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:45.490847 26898 raft_consensus.cc:374] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:45.490895 26900 raft_consensus.cc:729] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e12ad8453704daca7b51365d9394795, State: Initialized, Role: FOLLOWER
I20260810 15:01:45.490895 26898 raft_consensus.cc:729] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf8469ca73f4c1c91f839d2d2c6e782, State: Initialized, Role: FOLLOWER
I20260810 15:01:45.491008 26898 consensus_queue.cc:260] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.491008 26900 consensus_queue.cc:260] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.491264 26900 ts_tablet_manager.cc:1428] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:01:45.491264 26898 ts_tablet_manager.cc:1428] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:01:45.491323 26423 heartbeater.cc:503] Master 127.28.193.254:33301 was elected leader, sending a full tablet report...
I20260810 15:01:45.491602 26643 heartbeater.cc:503] Master 127.28.193.254:33301 was elected leader, sending a full tablet report...
W20260810 15:01:45.583420 26650 tablet.cc:2369] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:45.583716 26901 raft_consensus.cc:482] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:45.583859 26901 raft_consensus.cc:504] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.584259 26901 leader_election.cc:288] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf8469ca73f4c1c91f839d2d2c6e782 (127.28.193.194:42765), 0e12ad8453704daca7b51365d9394795 (127.28.193.193:46343)
I20260810 15:01:45.588234 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.584755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56813 (local address 127.28.193.193:46343)
0810 15:01:45.584805 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.584807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.584935 (+ 128us) server_negotiation.cc:407] Connection header received
0810 15:01:45.585034 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.585035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.585044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.585073 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:45.585278 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.585972 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587278 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.587829 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587910 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.587919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.588119 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.588162 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:01:45.588234 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.584548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56933 (local address 127.28.193.194:42765)
0810 15:01:45.584621 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.584624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.584636 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:01:45.584698 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.584699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.584709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.584824 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:45.585030 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.585789 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587278 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.587850 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587913 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.587919 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.588119 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.588162 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:01:45.588253 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.584459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:42765 (local address 127.28.193.195:56933)
0810 15:01:45.584551 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.584564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.584620 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.584887 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.584889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.584891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:45.584962 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.584965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.585883 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.585884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.587190 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.587193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.587934 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.588035 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.588039 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.588105 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.588155 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":7}
I20260810 15:01:45.588265 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.584581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:46343 (local address 127.28.193.195:56813)
0810 15:01:45.584882 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.584890 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:45.584959 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:45.585134 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:45.585135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:45.585137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:45.585203 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.585204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.586049 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.586049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.587190 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:45.587193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.587934 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.587935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:45.588035 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.588039 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:45.588105 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:01:45.588153 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":14,"thread_start_us":98,"threads_started":1}
I20260810 15:01:45.588654 26349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4824cc12a624e82961ac45cff165fbc" candidate_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e12ad8453704daca7b51365d9394795" is_pre_election: true
I20260810 15:01:45.588680 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4824cc12a624e82961ac45cff165fbc" candidate_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" is_pre_election: true
I20260810 15:01:45.588835 26349 raft_consensus.cc:2454] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63c796ad4de4f11b70abc5f7bc377d0 in term 0.
I20260810 15:01:45.588828 26575 raft_consensus.cc:2454] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d63c796ad4de4f11b70abc5f7bc377d0 in term 0.
I20260810 15:01:45.589080 26691 leader_election.cc:302] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e12ad8453704daca7b51365d9394795, d63c796ad4de4f11b70abc5f7bc377d0; no voters:
I20260810 15:01:45.589275 26901 raft_consensus.cc:2781] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:45.589319 26901 raft_consensus.cc:482] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:45.589340 26901 raft_consensus.cc:3037] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:45.590108 26901 raft_consensus.cc:504] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.590219 26901 leader_election.cc:288] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [CANDIDATE]: Term 1 election: Requested vote from peers 7bf8469ca73f4c1c91f839d2d2c6e782 (127.28.193.194:42765), 0e12ad8453704daca7b51365d9394795 (127.28.193.193:46343)
I20260810 15:01:45.590363 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4824cc12a624e82961ac45cff165fbc" candidate_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782"
I20260810 15:01:45.590404 26349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4824cc12a624e82961ac45cff165fbc" candidate_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e12ad8453704daca7b51365d9394795"
I20260810 15:01:45.590442 26575 raft_consensus.cc:3037] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:45.590497 26349 raft_consensus.cc:3037] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:45.591202 26575 raft_consensus.cc:2454] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63c796ad4de4f11b70abc5f7bc377d0 in term 1.
I20260810 15:01:45.591276 26349 raft_consensus.cc:2454] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d63c796ad4de4f11b70abc5f7bc377d0 in term 1.
I20260810 15:01:45.591373 26691 leader_election.cc:302] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf8469ca73f4c1c91f839d2d2c6e782, d63c796ad4de4f11b70abc5f7bc377d0; no voters:
I20260810 15:01:45.591491 26901 raft_consensus.cc:2781] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:45.591655 26901 raft_consensus.cc:686] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 1 LEADER]: Becoming Leader. State: Replica: d63c796ad4de4f11b70abc5f7bc377d0, State: Running, Role: LEADER
I20260810 15:01:45.591756 26901 consensus_queue.cc:237] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } }
I20260810 15:01:45.592561 26093 catalog_manager.cc:5495] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d63c796ad4de4f11b70abc5f7bc377d0 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d63c796ad4de4f11b70abc5f7bc377d0" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 46435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:45.626016 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.623708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.28.193.195:46435)
0810 15:01:45.623772 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:45.623774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:45.623782 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:01:45.623822 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:45.623823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:45.623841 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:45.623901 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:45.624032 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.624504 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.625262 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:45.625448 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:45.625504 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:45.625655 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:45.625789 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:45.625861 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:45.625898 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:01:45.625918 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:01:45.627837 26349 raft_consensus.cc:1261] T c4824cc12a624e82961ac45cff165fbc P 0e12ad8453704daca7b51365d9394795 [term 1 FOLLOWER]: Refusing update from remote peer d63c796ad4de4f11b70abc5f7bc377d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:45.627846 26575 raft_consensus.cc:1261] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 1 FOLLOWER]: Refusing update from remote peer d63c796ad4de4f11b70abc5f7bc377d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:45.628225 26910 consensus_queue.cc:1035] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e12ad8453704daca7b51365d9394795" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 46343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:45.628379 26901 consensus_queue.cc:1035] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf8469ca73f4c1c91f839d2d2c6e782" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:45.630708 26915 mvcc.cc:205] Tried to move back new op lower bound from 7316988336648859648 to 7316988336504524800. Current Snapshot: MvccSnapshot[applied={T|T < 7316988336648859648}]
I20260810 15:01:45.631070 26917 mvcc.cc:205] Tried to move back new op lower bound from 7316988336648859648 to 7316988336504524800. Current Snapshot: MvccSnapshot[applied={T|T < 7316988336648859648}]
I20260810 15:01:45.631946 26916 mvcc.cc:205] Tried to move back new op lower bound from 7316988336648859648 to 7316988336504524800. Current Snapshot: MvccSnapshot[applied={T|T < 7316988336648859648}]
I20260810 15:01:45.633159 26092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:45.633256 26092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:45.634020 26092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c77f6e484b44559af77b07531434c0b: Sending DeleteTablet for 3 replicas of tablet c4824cc12a624e82961ac45cff165fbc
I20260810 15:01:45.634338 26540 tablet_service.cc:1514] Processing DeleteTablet for tablet c4824cc12a624e82961ac45cff165fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:45 UTC) from {username='slave'} at 127.0.0.1:57474
I20260810 15:01:45.634366 26768 tablet_service.cc:1514] Processing DeleteTablet for tablet c4824cc12a624e82961ac45cff165fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:45 UTC) from {username='slave'} at 127.0.0.1:46056
I20260810 15:01:45.634378 26311 tablet_service.cc:1514] Processing DeleteTablet for tablet c4824cc12a624e82961ac45cff165fbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:45 UTC) from {username='slave'} at 127.0.0.1:33634
I20260810 15:01:45.634549 26928 tablet_replica.cc:330] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: stopping tablet replica
I20260810 15:01:45.634640 26928 raft_consensus.cc:2229] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:45.634742 26928 raft_consensus.cc:2258] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:45.634817 26931 tablet_replica.cc:330] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: stopping tablet replica
I20260810 15:01:45.634874 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 26204
I20260810 15:01:45.634907 26931 raft_consensus.cc:2229] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:01:45.634984 26931 raft_consensus.cc:2258] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:45.642210 26931 ts_tablet_manager.cc:1905] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:45.644241 26931 ts_tablet_manager.cc:1918] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:45.644326 26931 log.cc:1196] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/c4824cc12a624e82961ac45cff165fbc
I20260810 15:01:45.644569 26931 ts_tablet_manager.cc:1939] T c4824cc12a624e82961ac45cff165fbc P d63c796ad4de4f11b70abc5f7bc377d0: Deleting consensus metadata
I20260810 15:01:45.644624 26928 ts_tablet_manager.cc:1905] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:45.646085 26060 catalog_manager.cc:4841] TS d63c796ad4de4f11b70abc5f7bc377d0 (127.28.193.195:46435): tablet c4824cc12a624e82961ac45cff165fbc (table test-table [id=8614cfc6bc8e4f948ab418516f4c05a1]) successfully deleted
I20260810 15:01:45.646306 26928 ts_tablet_manager.cc:1918] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:45.646387 26928 log.cc:1196] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/c4824cc12a624e82961ac45cff165fbc
I20260810 15:01:45.646633 26928 ts_tablet_manager.cc:1939] T c4824cc12a624e82961ac45cff165fbc P 7bf8469ca73f4c1c91f839d2d2c6e782: Deleting consensus metadata
I20260810 15:01:45.647078 26067 catalog_manager.cc:4841] TS 7bf8469ca73f4c1c91f839d2d2c6e782 (127.28.193.194:42765): tablet c4824cc12a624e82961ac45cff165fbc (table test-table [id=8614cfc6bc8e4f948ab418516f4c05a1]) successfully deleted
W20260810 15:01:45.648185 26067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:46343 (error 108)
I20260810 15:01:45.648667 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 26429
I20260810 15:01:45.648671 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:45.648531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:46343 (local address 127.0.0.1:33650)
0810 15:01:45.648589 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:45.648634 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 15:01:45.652122 26067 catalog_manager.cc:4584] TS 0e12ad8453704daca7b51365d9394795 (127.28.193.193:46343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4824cc12a624e82961ac45cff165fbc: Network error: Client connection negotiation failed: client connection to 127.28.193.193:46343: connect: Connection refused (error 111)
I20260810 15:01:45.661042 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 26653
I20260810 15:01:45.667358 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 26031
2026-08-10T15:01:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:45 dist-test-slave-cvwc krb5kdc[26942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:45 dist-test-slave-cvwc krb5kdc[26942](info): set up 2 sockets
Aug 10 15:01:45 dist-test-slave-cvwc krb5kdc[26942](info): commencing operation
krb5kdc: starting...
W20260810 15:01:46.811147 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:46 dist-test-slave-cvwc krb5kdc[26942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:01:46.930292 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44753
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:47.010066 26966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:47.010267 26966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:47.010288 26966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:47.011389 26966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:47.011428 26966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:47.011440 26966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:47.011451 26966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:47.011462 26966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:47.012910 26966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:47.013123 26966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:47.013338 26966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:47.018002 26972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.017961 26977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.017983 26966 server_base.cc:1034] running on GCE node
W20260810 15:01:47.017959 26973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.018378 26966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:47.018652 26966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:47.019829 26966 hybrid_clock.cc:648] HybridClock initialized: now 1786374107019813 us; error 27 us; skew 500 ppm
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:47.022374 26966 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:01:47.022827 26966 webserver.cc:460] Webserver started at http://127.28.193.254:44985/ using document root <none> and password file <none>
I20260810 15:01:47.023026 26966 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:47.023077 26966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:47.023223 26966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:47.024258 26966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "cb4f92daa7aa4bd098388f1152464994"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "d823c5b589454a047ec2e083050f0b3d"
server_key_iv: "b9ce62e3025f475873e7cf2598af8eb8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.024644 26966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "cb4f92daa7aa4bd098388f1152464994"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "d823c5b589454a047ec2e083050f0b3d"
server_key_iv: "b9ce62e3025f475873e7cf2598af8eb8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.026008 26966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:47.026681 26990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.026844 26966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:47.026933 26966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "cb4f92daa7aa4bd098388f1152464994"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "d823c5b589454a047ec2e083050f0b3d"
server_key_iv: "b9ce62e3025f475873e7cf2598af8eb8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.027001 26966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:47.047537 26966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:47.049295 26966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:47.049451 26966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:47.053566 26966 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:44753
I20260810 15:01:47.053610 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:44753 every 8 connection(s)
I20260810 15:01:47.053992 26966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:47.054853 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:47.054848 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 26966
I20260810 15:01:47.054970 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:47.055162 29447 external_mini_cluster.cc:1428] Setting key f209ef9fa36f602e54e8caa92f252117
I20260810 15:01:47.058105 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: Bootstrap starting.
I20260810 15:01:47.058794 27068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:47.059057 27068 log.cc:824] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:47.059671 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: No bootstrap required, opened a new log
I20260810 15:01:47.061395 27068 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } }
I20260810 15:01:47.061529 27068 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:47.061560 27068 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4f92daa7aa4bd098388f1152464994, State: Initialized, Role: FOLLOWER
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:47.061733 27068 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } }
I20260810 15:01:47.061828 27068 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:47.061859 27068 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:47.061908 27068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:47.062489 27068 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } }
I20260810 15:01:47.062582 27068 leader_election.cc:302] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb4f92daa7aa4bd098388f1152464994; no voters:
I20260810 15:01:47.062731 27068 leader_election.cc:288] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:47.062777 27077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:47.062907 27077 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [term 1 LEADER]: Becoming Leader. State: Replica: cb4f92daa7aa4bd098388f1152464994, State: Running, Role: LEADER
I20260810 15:01:47.062990 27077 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } }
I20260810 15:01:47.062996 27068 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:47.064141 27081 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb4f92daa7aa4bd098388f1152464994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } } }
I20260810 15:01:47.064239 27081 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:47.064249 27078 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb4f92daa7aa4bd098388f1152464994. Latest consensus state: current_term: 1 leader_uuid: "cb4f92daa7aa4bd098388f1152464994" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb4f92daa7aa4bd098388f1152464994" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 44753 } } }
I20260810 15:01:47.064416 27078 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994 [sys.catalog]: This master's current role is: LEADER
W20260810 15:01:47.064729 27099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:01:47.064787 27099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:01:47.064841 27100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:47.064988 27100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:47.065696 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.055665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.28.193.254:44753)
0810 15:01:47.055894 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.055904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.055935 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:01:47.056366 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.056382 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.056447 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.056598 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:47.057188 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.058542 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.059320 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.059720 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.059841 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.062232 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:47.062236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:47.062236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:47.062246 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:47.063928 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.064326 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.064329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.064329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.064433 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.064674 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.064675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.064676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.064985 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:47.065122 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.065384 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.065528 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":113,"threads_started":1}
I20260810 15:01:47.070439 27100 catalog_manager.cc:1347] Generated new cluster ID: 0ee6ae738fea4c2cb7f2ca35cee82b69
I20260810 15:01:47.070490 27100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:47.077452 27100 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:47.077986 27100 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:47.087553 27100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: Generated new TSK 0
I20260810 15:01:47.087728 27100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:01:47.110675 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:47.191448 27114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:47.191642 27114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:47.191663 27114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:47.192780 27114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:47.192819 27114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:47.192844 27114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:47.194314 27114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:47.194536 27114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:47.194767 27114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:47.199524 27120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.199648 27114 server_base.cc:1034] running on GCE node
W20260810 15:01:47.199553 27119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.199537 27123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.199916 27114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:47.200129 27114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:47.201270 27114 hybrid_clock.cc:648] HybridClock initialized: now 1786374107201254 us; error 31 us; skew 500 ppm
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:47.203608 27114 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:01:47.204000 27114 webserver.cc:460] Webserver started at http://127.28.193.193:36693/ using document root <none> and password file <none>
I20260810 15:01:47.204191 27114 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:47.204239 27114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:47.204367 27114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:47.205493 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "dc8c78e5f72844a88a9f75dc9efe4e94"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "40617e559e31f843918ef0dc1633fbee"
server_key_iv: "919557329ffa2849deeb7bb30b47ee34"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.205885 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "dc8c78e5f72844a88a9f75dc9efe4e94"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "40617e559e31f843918ef0dc1633fbee"
server_key_iv: "919557329ffa2849deeb7bb30b47ee34"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.207123 27114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:47.207770 27137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.207916 27114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:47.207998 27114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "dc8c78e5f72844a88a9f75dc9efe4e94"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "40617e559e31f843918ef0dc1633fbee"
server_key_iv: "919557329ffa2849deeb7bb30b47ee34"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.208065 27114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:47.218585 27114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:47.220439 27114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:47.220568 27114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:47.220779 27114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:47.221073 27114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:47.221107 27114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.221135 27114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:47.221158 27114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.226616 27114 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:36047
I20260810 15:01:47.226673 27316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:36047 every 8 connection(s)
I20260810 15:01:47.226948 27114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:47.235193 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 27114
I20260810 15:01:47.235318 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:47.235464 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.227601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:57417 (local address 127.28.193.254:44753)
0810 15:01:47.227663 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.227665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.227854 (+ 189us) server_negotiation.cc:407] Connection header received
0810 15:01:47.228332 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.228334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.228344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.228380 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:47.229269 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.229871 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.230832 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.231035 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.231088 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.233058 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:47.233060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:47.233060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:47.233067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:47.234253 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.234613 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.234613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.234614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.234664 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.234861 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.234862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.234862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.234960 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:47.235004 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.235278 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.235369 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:01:47.235513 29447 external_mini_cluster.cc:1428] Setting key 6a4b547fb41bd269bba4daf63c19d1c4
I20260810 15:01:47.235626 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.227555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44753 (local address 127.28.193.193:57417)
0810 15:01:47.227712 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.227774 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.228231 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.228440 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.228444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.229003 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:47.229186 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.229191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.229953 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.229954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.230736 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.230739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.231112 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.231113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.231214 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.231665 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:47.231673 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:47.232956 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:47.234335 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.234341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.234343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.234553 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.234711 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.234712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.234712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.234817 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.235010 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:47.235016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:47.235141 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.235229 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.235376 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260810 15:01:47.236096 27317 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44753
I20260810 15:01:47.236194 27317 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:47.236379 27317 heartbeater.cc:511] Master 127.28.193.254:44753 requested a full tablet report, sending...
I20260810 15:01:47.236904 27014 ts_manager.cc:194] Registered new tserver with Master: dc8c78e5f72844a88a9f75dc9efe4e94 (127.28.193.193:36047)
I20260810 15:01:47.237483 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:57417
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:01:47.259759 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:47.340847 27326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:47.341056 27326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:47.341078 27326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:47.342233 27326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:47.342272 27326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:47.342300 27326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:47.343716 27326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:47.343951 27326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:47.344167 27326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:47.349047 27333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.349025 27339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.349025 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.349140 27326 server_base.cc:1034] running on GCE node
I20260810 15:01:47.349462 27326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:47.349650 27326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:47.350813 27326 hybrid_clock.cc:648] HybridClock initialized: now 1786374107350809 us; error 40 us; skew 500 ppm
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:47.353390 27326 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:01:47.353849 27326 webserver.cc:460] Webserver started at http://127.28.193.194:44621/ using document root <none> and password file <none>
I20260810 15:01:47.354036 27326 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:47.354096 27326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:47.354213 27326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:47.355265 27326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "87d24e87e1544da8ba8d390bdb2a0190"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "10289dd866f53ab5f0d3b3f4aadb393e"
server_key_iv: "5e6baa12532c164dac7ca7466c642be8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.355623 27326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "87d24e87e1544da8ba8d390bdb2a0190"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "10289dd866f53ab5f0d3b3f4aadb393e"
server_key_iv: "5e6baa12532c164dac7ca7466c642be8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.356949 27326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:47.357642 27349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.357805 27326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:47.357892 27326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "87d24e87e1544da8ba8d390bdb2a0190"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "10289dd866f53ab5f0d3b3f4aadb393e"
server_key_iv: "5e6baa12532c164dac7ca7466c642be8"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.357949 27326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:47.367527 27326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:47.369300 27326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:47.369407 27326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:47.369597 27326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:47.369907 27326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:47.369942 27326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.369962 27326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:47.369973 27326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.375396 27326 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:40081
I20260810 15:01:47.375465 27529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:40081 every 8 connection(s)
I20260810 15:01:47.375749 27326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:47.384048 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.376472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:42007 (local address 127.28.193.254:44753)
0810 15:01:47.376524 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.376526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.376743 (+ 217us) server_negotiation.cc:407] Connection header received
0810 15:01:47.377229 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.377230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.377238 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.377301 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:47.378288 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.378782 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.379699 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.379880 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.379924 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.381938 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:47.381939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:47.381939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:47.381946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:47.383025 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.383363 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.383363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.383364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.383411 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.383582 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.383583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.383583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.383675 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:47.383718 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.383922 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.383981 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:01:47.384202 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.376458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44753 (local address 127.28.193.194:42007)
0810 15:01:47.376623 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.376665 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.377152 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.377371 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.377375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.378017 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:47.378213 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.378219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.378856 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.378856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.379640 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.379643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.379935 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.379935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.380038 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.380512 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:47.380521 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:47.381843 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:47.383089 (+ 1246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.383095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.383097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.383309 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.383455 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.383456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.383456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.383542 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.383715 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:47.383717 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:47.383823 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.383904 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.384019 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260810 15:01:47.384680 27530 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44753
I20260810 15:01:47.384698 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 27326
I20260810 15:01:47.384771 27530 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:47.384794 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:47.384954 27530 heartbeater.cc:511] Master 127.28.193.254:44753 requested a full tablet report, sending...
I20260810 15:01:47.385004 29447 external_mini_cluster.cc:1428] Setting key 3a02b7f24cdf109fdaf999de80f11314
I20260810 15:01:47.385337 27014 ts_manager.cc:194] Registered new tserver with Master: 87d24e87e1544da8ba8d390bdb2a0190 (127.28.193.194:40081)
I20260810 15:01:47.385854 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:42007
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:01:47.408509 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:45875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:47.489282 27541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:47.489485 27541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:47.489516 27541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:47.490707 27541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:47.490749 27541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:47.490784 27541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:47.492192 27541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:45875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:44753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:47.492431 27541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:47.492647 27541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:47.497354 27548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.497382 27552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.497372 27550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.497521 27541 server_base.cc:1034] running on GCE node
I20260810 15:01:47.497788 27541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:47.497994 27541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:47.499132 27541 hybrid_clock.cc:648] HybridClock initialized: now 1786374107499123 us; error 26 us; skew 500 ppm
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:47.501384 27541 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:01:47.501811 27541 webserver.cc:460] Webserver started at http://127.28.193.195:40383/ using document root <none> and password file <none>
I20260810 15:01:47.502014 27541 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:47.502060 27541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:47.502182 27541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:47.503213 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "f9323a3b14b84b639a462c9cf9a59ae4"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "e5ca5ee6705c28b88c1f0637111342fa"
server_key_iv: "ff215d10f1f83d5dbdd94c3f2591e6dc"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.503541 27541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "f9323a3b14b84b639a462c9cf9a59ae4"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "e5ca5ee6705c28b88c1f0637111342fa"
server_key_iv: "ff215d10f1f83d5dbdd94c3f2591e6dc"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.504745 27541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:47.505455 27562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.505613 27541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:47.505693 27541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "f9323a3b14b84b639a462c9cf9a59ae4"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "e5ca5ee6705c28b88c1f0637111342fa"
server_key_iv: "ff215d10f1f83d5dbdd94c3f2591e6dc"
server_key_version: "encryptionkey@0"
I20260810 15:01:47.505788 27541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:47.516348 27541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:47.518343 27541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:47.518489 27541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:47.518700 27541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:47.518993 27541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:47.519027 27541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.519054 27541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:47.519078 27541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:47.524343 27541 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:35181
I20260810 15:01:47.524402 27759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:35181 every 8 connection(s)
I20260810 15:01:47.524672 27541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:01:47 dist-test-slave-cvwc krb5kdc[26942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374107, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:47.532878 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.525340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:53273 (local address 127.28.193.254:44753)
0810 15:01:47.525404 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.525405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.525567 (+ 162us) server_negotiation.cc:407] Connection header received
0810 15:01:47.526094 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.526095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.526104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.526144 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:47.527013 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.527508 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.528483 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.528673 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.528724 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.530730 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:47.530732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:47.530732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:47.530738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:47.531793 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.532130 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.532131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.532131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.532173 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.532418 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.532418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.532419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.532512 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:47.532555 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.532754 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.532810 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:01:47.533071 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.525298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:44753 (local address 127.28.193.195:53273)
0810 15:01:47.525453 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.525492 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.525993 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.526201 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.526204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.526739 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:47.526937 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.526942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.527583 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.527584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.528412 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.528415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.528730 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.528730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.528841 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.529291 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:47.529299 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:47.530639 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:47.531865 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.531872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.531873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.532074 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.532244 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:47.532245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:47.532246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:47.532365 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:47.532554 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:47.532557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:47.532662 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.532739 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.532864 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260810 15:01:47.533288 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 27541
I20260810 15:01:47.533401 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:47.533578 27760 heartbeater.cc:347] Connected to a master server at 127.28.193.254:44753
I20260810 15:01:47.533601 29447 external_mini_cluster.cc:1428] Setting key cfe074cc5a760292a6352c1d3b3968d0
I20260810 15:01:47.533677 27760 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:47.533877 27760 heartbeater.cc:511] Master 127.28.193.254:44753 requested a full tablet report, sending...
I20260810 15:01:47.534215 27014 ts_manager.cc:194] Registered new tserver with Master: f9323a3b14b84b639a462c9cf9a59ae4 (127.28.193.195:35181)
I20260810 15:01:47.534641 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:47.534670 27014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:53273
I20260810 15:01:47.541489 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.536023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.28.193.254:44753)
0810 15:01:47.536057 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.536059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.536170 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:01:47.536230 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.536231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.536240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.536284 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:47.536801 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.537265 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.538018 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.538205 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.538253 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.539443 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:47.539445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:47.539445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:47.539452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:47.540393 (+ 941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.540738 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.540739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.540739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.540788 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:47.540984 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:47.540984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:47.540985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:47.541111 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:47.541202 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.541328 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.541398 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:01:47.542886 27014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:47.544371 27014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:47.550490 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:40081 (local address 127.0.0.1:44914)
0810 15:01:47.547122 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.547128 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.547159 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.547398 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.547399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.547401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:47.547443 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.547445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548301 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.548302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.549380 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.549383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550226 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.550304 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550318 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.550382 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.550432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260810 15:01:47.550601 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:35181 (local address 127.0.0.1:50514)
0810 15:01:47.546947 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.546985 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.547047 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.547222 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.547226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.547228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:47.547329 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.547331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548644 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.548645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.549473 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.549476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550359 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.550422 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550426 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.550485 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.550527 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":63,"threads_started":1}
I20260810 15:01:47.550817 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.28.193.194:40081)
0810 15:01:47.546949 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.546953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.547192 (+ 239us) server_negotiation.cc:407] Connection header received
0810 15:01:47.547233 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.547235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.547250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.547318 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:47.547520 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548208 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.549470 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550151 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550212 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550276 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.550679 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.550723 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260810 15:01:47.550874 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.28.193.195:35181)
0810 15:01:47.546777 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.546782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.547036 (+ 254us) server_negotiation.cc:407] Connection header received
0810 15:01:47.547104 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.547106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.547119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.547198 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:47.547423 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548533 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.549570 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550211 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550279 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550325 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.550747 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.550790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260810 15:01:47.551004 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36047 (local address 127.0.0.1:41920)
0810 15:01:47.547211 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.547219 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.547257 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.547534 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.547535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.547537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:47.547573 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.547574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548777 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.548778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.549859 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.549862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550786 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.550845 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550849 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.550896 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.550931 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260810 15:01:47.551260 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.546772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.28.193.193:36047)
0810 15:01:47.546950 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.546953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.547308 (+ 355us) server_negotiation.cc:407] Connection header received
0810 15:01:47.547352 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.547354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.547371 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.547435 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:47.547645 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.548677 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.549958 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.550639 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.550702 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.550753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.551159 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.551201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260810 15:01:47.551473 27420 tablet_service.cc:1467] Processing CreateTablet for tablet 4596e8b7005f4a09864d92db03aa0471 (DEFAULT_TABLE table=test-table [id=81d879896d36467383cf101678edbb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:47.551478 27643 tablet_service.cc:1467] Processing CreateTablet for tablet 4596e8b7005f4a09864d92db03aa0471 (DEFAULT_TABLE table=test-table [id=81d879896d36467383cf101678edbb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:47.551820 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596e8b7005f4a09864d92db03aa0471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:47.551820 27643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596e8b7005f4a09864d92db03aa0471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:47.551891 27217 tablet_service.cc:1467] Processing CreateTablet for tablet 4596e8b7005f4a09864d92db03aa0471 (DEFAULT_TABLE table=test-table [id=81d879896d36467383cf101678edbb2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:47.552168 27217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4596e8b7005f4a09864d92db03aa0471. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:47.555034 27789 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Bootstrap starting.
I20260810 15:01:47.555274 27785 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Bootstrap starting.
I20260810 15:01:47.555644 27789 tablet_bootstrap.cc:654] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:47.555827 27785 tablet_bootstrap.cc:654] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:47.556130 27789 log.cc:824] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:47.556270 27785 log.cc:824] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:47.556457 27791 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Bootstrap starting.
I20260810 15:01:47.556879 27789 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: No bootstrap required, opened a new log
I20260810 15:01:47.556951 27789 ts_tablet_manager.cc:1397] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:47.557017 27785 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: No bootstrap required, opened a new log
I20260810 15:01:47.557080 27785 ts_tablet_manager.cc:1397] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:47.558337 27791 tablet_bootstrap.cc:654] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:47.558559 27789 raft_consensus.cc:348] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.558717 27789 raft_consensus.cc:374] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:47.558751 27789 raft_consensus.cc:729] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d24e87e1544da8ba8d390bdb2a0190, State: Initialized, Role: FOLLOWER
I20260810 15:01:47.558895 27791 log.cc:824] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:47.558892 27789 consensus_queue.cc:260] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.558851 27785 raft_consensus.cc:348] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.559020 27785 raft_consensus.cc:374] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:47.559056 27785 raft_consensus.cc:729] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8c78e5f72844a88a9f75dc9efe4e94, State: Initialized, Role: FOLLOWER
I20260810 15:01:47.559159 27789 ts_tablet_manager.cc:1428] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:47.559294 27530 heartbeater.cc:503] Master 127.28.193.254:44753 was elected leader, sending a full tablet report...
I20260810 15:01:47.559263 27785 consensus_queue.cc:260] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.559597 27785 ts_tablet_manager.cc:1428] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:01:47.559758 27317 heartbeater.cc:503] Master 127.28.193.254:44753 was elected leader, sending a full tablet report...
I20260810 15:01:47.560623 27791 tablet_bootstrap.cc:492] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: No bootstrap required, opened a new log
I20260810 15:01:47.560688 27791 ts_tablet_manager.cc:1397] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 15:01:47.562157 27791 raft_consensus.cc:348] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.562342 27791 raft_consensus.cc:374] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:47.562397 27791 raft_consensus.cc:729] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9323a3b14b84b639a462c9cf9a59ae4, State: Initialized, Role: FOLLOWER
I20260810 15:01:47.562513 27791 consensus_queue.cc:260] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.562804 27791 ts_tablet_manager.cc:1428] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:47.562831 27760 heartbeater.cc:503] Master 127.28.193.254:44753 was elected leader, sending a full tablet report...
W20260810 15:01:47.626318 27535 tablet.cc:2369] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:01:47.727706 27318 tablet.cc:2369] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:01:47.775228 27762 tablet.cc:2369] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:47.799443 27800 raft_consensus.cc:482] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:47.799564 27800 raft_consensus.cc:504] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.799901 27800 leader_election.cc:288] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc8c78e5f72844a88a9f75dc9efe4e94 (127.28.193.193:36047), 87d24e87e1544da8ba8d390bdb2a0190 (127.28.193.194:40081)
I20260810 15:01:47.802719 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.800204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:39801 (local address 127.28.193.193:36047)
0810 15:01:47.800245 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.800247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.800356 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:01:47.800380 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.800381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.800390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.800430 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:47.800630 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.801094 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802049 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.802400 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802461 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.802470 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.802607 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.802649 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:01:47.802760 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.800079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:40081 (local address 127.28.193.195:41395)
0810 15:01:47.800133 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.800146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.800197 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.800377 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.800377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.800379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:47.800475 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.800477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.801132 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.801133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.801955 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.801958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.802474 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.802553 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.802555 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.802600 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.802637 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":44}
I20260810 15:01:47.802723 27784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.800196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:41395 (local address 127.28.193.194:40081)
0810 15:01:47.800254 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.800256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.800264 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:01:47.800294 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.800295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.800303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.800364 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:47.800557 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.801047 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802041 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.802407 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802470 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.802478 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.802619 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.802656 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:01:47.802779 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.800091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36047 (local address 127.28.193.195:39801)
0810 15:01:47.800269 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.800278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:47.800322 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:47.800448 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:47.800449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:47.800450 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:47.800517 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.800519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.801177 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.801178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.801966 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:47.801969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.802473 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.802474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:47.802546 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.802549 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:47.802595 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:01:47.802635 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260810 15:01:47.803082 27460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596e8b7005f4a09864d92db03aa0471" candidate_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d24e87e1544da8ba8d390bdb2a0190" is_pre_election: true
I20260810 15:01:47.803239 27460 raft_consensus.cc:2454] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9323a3b14b84b639a462c9cf9a59ae4 in term 0.
I20260810 15:01:47.803234 27246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596e8b7005f4a09864d92db03aa0471" candidate_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" is_pre_election: true
I20260810 15:01:47.803387 27246 raft_consensus.cc:2454] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9323a3b14b84b639a462c9cf9a59ae4 in term 0.
I20260810 15:01:47.803455 27565 leader_election.cc:302] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d24e87e1544da8ba8d390bdb2a0190, f9323a3b14b84b639a462c9cf9a59ae4; no voters:
I20260810 15:01:47.803570 27800 raft_consensus.cc:2781] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:47.803622 27800 raft_consensus.cc:482] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:47.803643 27800 raft_consensus.cc:3037] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:47.804292 27800 raft_consensus.cc:504] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.804395 27800 leader_election.cc:288] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [CANDIDATE]: Term 1 election: Requested vote from peers dc8c78e5f72844a88a9f75dc9efe4e94 (127.28.193.193:36047), 87d24e87e1544da8ba8d390bdb2a0190 (127.28.193.194:40081)
I20260810 15:01:47.804549 27246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596e8b7005f4a09864d92db03aa0471" candidate_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94"
I20260810 15:01:47.804570 27460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4596e8b7005f4a09864d92db03aa0471" candidate_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87d24e87e1544da8ba8d390bdb2a0190"
I20260810 15:01:47.804641 27460 raft_consensus.cc:3037] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:47.804641 27246 raft_consensus.cc:3037] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:47.805500 27246 raft_consensus.cc:2454] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9323a3b14b84b639a462c9cf9a59ae4 in term 1.
I20260810 15:01:47.805500 27460 raft_consensus.cc:2454] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9323a3b14b84b639a462c9cf9a59ae4 in term 1.
I20260810 15:01:47.805655 27565 leader_election.cc:302] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d24e87e1544da8ba8d390bdb2a0190, f9323a3b14b84b639a462c9cf9a59ae4; no voters:
I20260810 15:01:47.805779 27800 raft_consensus.cc:2781] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:47.805913 27800 raft_consensus.cc:686] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 1 LEADER]: Becoming Leader. State: Replica: f9323a3b14b84b639a462c9cf9a59ae4, State: Running, Role: LEADER
I20260810 15:01:47.805986 27800 consensus_queue.cc:237] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } }
I20260810 15:01:47.806818 27014 catalog_manager.cc:5495] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9323a3b14b84b639a462c9cf9a59ae4 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9323a3b14b84b639a462c9cf9a59ae4" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 35181 } health_report { overall_health: HEALTHY } } }
I20260810 15:01:47.832178 27781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.830084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.28.193.195:35181)
0810 15:01:47.830138 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:47.830140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:47.830149 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:01:47.830184 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:47.830185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:47.830204 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:47.830257 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:47.830396 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.830895 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.831604 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:47.831792 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:47.831838 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:47.831936 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:47.832014 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:47.832076 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:47.832100 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 15:01:47.832108 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:01:47.833832 27460 raft_consensus.cc:1261] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 1 FOLLOWER]: Refusing update from remote peer f9323a3b14b84b639a462c9cf9a59ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:47.833840 27246 raft_consensus.cc:1261] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 1 FOLLOWER]: Refusing update from remote peer f9323a3b14b84b639a462c9cf9a59ae4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:47.834133 27800 consensus_queue.cc:1035] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8c78e5f72844a88a9f75dc9efe4e94" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 36047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:47.834285 27803 consensus_queue.cc:1035] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87d24e87e1544da8ba8d390bdb2a0190" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 40081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:47.836433 27808 mvcc.cc:205] Tried to move back new op lower bound from 7316988345684795392 to 7316988345573793792. Current Snapshot: MvccSnapshot[applied={T|T < 7316988345684795392}]
I20260810 15:01:47.837096 27809 mvcc.cc:205] Tried to move back new op lower bound from 7316988345684795392 to 7316988345573793792. Current Snapshot: MvccSnapshot[applied={T|T < 7316988345684795392}]
I20260810 15:01:47.837472 27810 mvcc.cc:205] Tried to move back new op lower bound from 7316988345684795392 to 7316988345573793792. Current Snapshot: MvccSnapshot[applied={T|T < 7316988345684795392}]
I20260810 15:01:47.839649 27012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:47.839731 27012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:47.840441 27012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb4f92daa7aa4bd098388f1152464994: Sending DeleteTablet for 3 replicas of tablet 4596e8b7005f4a09864d92db03aa0471
I20260810 15:01:47.840744 27420 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596e8b7005f4a09864d92db03aa0471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:44914
I20260810 15:01:47.840775 27643 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596e8b7005f4a09864d92db03aa0471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:50514
I20260810 15:01:47.840796 27217 tablet_service.cc:1514] Processing DeleteTablet for tablet 4596e8b7005f4a09864d92db03aa0471 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:47 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:41920
I20260810 15:01:47.841022 27822 tablet_replica.cc:330] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: stopping tablet replica
I20260810 15:01:47.841118 27822 raft_consensus.cc:2229] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:47.841207 27822 raft_consensus.cc:2258] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:47.841413 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 27114
I20260810 15:01:47.841584 27822 ts_tablet_manager.cc:1905] T 4596e8b7005f4a09864d92db03aa0471 P dc8c78e5f72844a88a9f75dc9efe4e94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:47.849833 27821 tablet_replica.cc:330] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: stopping tablet replica
I20260810 15:01:47.849942 27821 raft_consensus.cc:2229] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:01:47.850061 27821 raft_consensus.cc:2258] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:47.850214 27818 tablet_replica.cc:330] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: stopping tablet replica
I20260810 15:01:47.850328 27818 raft_consensus.cc:2229] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:47.850438 27818 raft_consensus.cc:2258] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:47.850888 27818 ts_tablet_manager.cc:1905] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:47.852468 27818 ts_tablet_manager.cc:1918] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:47.852541 27818 log.cc:1196] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/4596e8b7005f4a09864d92db03aa0471
I20260810 15:01:47.852807 27818 ts_tablet_manager.cc:1939] T 4596e8b7005f4a09864d92db03aa0471 P 87d24e87e1544da8ba8d390bdb2a0190: Deleting consensus metadata
W20260810 15:01:47.852999 26991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:36047 (error 108)
I20260810 15:01:47.853195 27821 ts_tablet_manager.cc:1905] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:47.853204 26995 catalog_manager.cc:4841] TS 87d24e87e1544da8ba8d390bdb2a0190 (127.28.193.194:40081): tablet 4596e8b7005f4a09864d92db03aa0471 (table test-table [id=81d879896d36467383cf101678edbb2e]) successfully deleted
I20260810 15:01:47.853546 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 27326
I20260810 15:01:47.855950 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:47.853640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:36047 (local address 127.0.0.1:41930)
0810 15:01:47.855864 (+ 2224us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:47.855913 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2199}
W20260810 15:01:47.856228 26991 catalog_manager.cc:4584] TS dc8c78e5f72844a88a9f75dc9efe4e94 (127.28.193.193:36047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4596e8b7005f4a09864d92db03aa0471: Network error: Client connection negotiation failed: client connection to 127.28.193.193:36047: connect: Connection refused (error 111)
I20260810 15:01:47.857828 27821 ts_tablet_manager.cc:1918] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:47.857909 27821 log.cc:1196] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/4596e8b7005f4a09864d92db03aa0471
I20260810 15:01:47.858181 27821 ts_tablet_manager.cc:1939] T 4596e8b7005f4a09864d92db03aa0471 P f9323a3b14b84b639a462c9cf9a59ae4: Deleting consensus metadata
I20260810 15:01:47.858625 26997 catalog_manager.cc:4841] TS f9323a3b14b84b639a462c9cf9a59ae4 (127.28.193.195:35181): tablet 4596e8b7005f4a09864d92db03aa0471 (table test-table [id=81d879896d36467383cf101678edbb2e]) successfully deleted
I20260810 15:01:47.861008 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 27541
I20260810 15:01:47.875140 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 26966
2026-08-10T15:01:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2212 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T15:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:47Z Disabled control of system clock
I20260810 15:01:47.907867 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42775
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:47.987555 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:47.987762 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:47.987782 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:47.988902 27833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:47.988943 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:47.988955 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:47.988966 27833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:47.988976 27833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:47.990314 27833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:42775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:47.990542 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:47.990819 27833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:47.995349 27847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:47.995357 27842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.995496 27833 server_base.cc:1034] running on GCE node
W20260810 15:01:47.995473 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:47.995893 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:47.996165 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:47.997304 27833 hybrid_clock.cc:648] HybridClock initialized: now 1786374107997290 us; error 25 us; skew 500 ppm
I20260810 15:01:47.998445 27833 webserver.cc:460] Webserver started at http://127.28.193.254:45691/ using document root <none> and password file <none>
I20260810 15:01:47.998670 27833 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:47.998718 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:47.998862 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:47.999882 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "8c9d5b448c974013ae3ca9169dcc5b86"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "8cf632df7d2d4bbe631544b3d106406e"
server_key_iv: "e33fe075395112ee2093bab0fff90bc2"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.000211 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "8c9d5b448c974013ae3ca9169dcc5b86"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "8cf632df7d2d4bbe631544b3d106406e"
server_key_iv: "e33fe075395112ee2093bab0fff90bc2"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.001395 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:48.002141 27858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.002292 27833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.002374 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "8c9d5b448c974013ae3ca9169dcc5b86"
format_stamp: "Formatted at 2026-08-10 15:01:47 on dist-test-slave-cvwc"
server_key: "8cf632df7d2d4bbe631544b3d106406e"
server_key_iv: "e33fe075395112ee2093bab0fff90bc2"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.002440 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:48.013207 27833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:48.013449 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:48.013563 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:48.017197 27833 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:42775
I20260810 15:01:48.017226 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:42775 every 8 connection(s)
I20260810 15:01:48.017517 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:48.018486 27937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:48.020591 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: Bootstrap starting.
I20260810 15:01:48.021117 27937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:48.021345 27937 log.cc:824] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:48.021948 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 27833
I20260810 15:01:48.021976 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: No bootstrap required, opened a new log
I20260810 15:01:48.022040 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:48.022215 29447 external_mini_cluster.cc:1428] Setting key a6dc18f557076194493f6e99fb2c6a44
I20260810 15:01:48.023844 27937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } }
I20260810 15:01:48.024006 27937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:48.024052 27937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d5b448c974013ae3ca9169dcc5b86, State: Initialized, Role: FOLLOWER
I20260810 15:01:48.024178 27937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } }
I20260810 15:01:48.024252 27937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:48.024283 27937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:48.024333 27937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:48.024823 27937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } }
I20260810 15:01:48.024915 27937 leader_election.cc:302] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9d5b448c974013ae3ca9169dcc5b86; no voters:
I20260810 15:01:48.025071 27937 leader_election.cc:288] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:48.025113 27947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:48.025223 27947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9d5b448c974013ae3ca9169dcc5b86, State: Running, Role: LEADER
I20260810 15:01:48.025319 27937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:48.025310 27947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } }
I20260810 15:01:48.025619 27947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c9d5b448c974013ae3ca9169dcc5b86. Latest consensus state: current_term: 1 leader_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } } }
I20260810 15:01:48.025691 27947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:48.025928 27954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:48.026446 27954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:48.026671 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d5b448c974013ae3ca9169dcc5b86" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 42775 } } }
I20260810 15:01:48.026746 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:48.028301 27954 catalog_manager.cc:1347] Generated new cluster ID: 7f694064de984a2ba3ccd4354bb1b939
I20260810 15:01:48.028353 27954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:48.028674 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.022685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.28.193.254:42775)
0810 15:01:48.022879 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.022887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.022910 (+ 23us) server_negotiation.cc:407] Connection header received
0810 15:01:48.023740 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.023746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.023817 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.023927 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:48.024183 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.025517 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.026485 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.027009 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.027121 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.027787 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:48.027791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:48.027791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:48.028006 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:48.028045 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:48.028059 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:48.028118 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.028390 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.028472 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260810 15:01:48.034044 27954 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:48.034750 27954 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:48.039386 27954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: Generated new TSK 0
I20260810 15:01:48.039541 27954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:48.046600 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--builtin_ntp_servers=127.28.193.212:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:48.145629 27976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:48.145867 27976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:48.145897 27976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:48.147010 27976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:48.147046 27976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:48.147078 27976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:48.148717 27976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:48.148975 27976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:48.149242 27976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:48.154027 27984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.154178 27987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.154044 27985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:48.154580 27976 server_base.cc:1034] running on GCE node
I20260810 15:01:48.154790 27976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:48.155006 27976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:48.156139 27976 hybrid_clock.cc:648] HybridClock initialized: now 1786374108156126 us; error 30 us; skew 500 ppm
I20260810 15:01:48.157305 27976 webserver.cc:460] Webserver started at http://127.28.193.193:40369/ using document root <none> and password file <none>
I20260810 15:01:48.157536 27976 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:48.157584 27976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:48.157735 27976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:48.158828 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "c6f61309d827430c81aece8062159bfb"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "7c490df1fcb3267527913b2b3ef4aaf3"
server_key_iv: "180d3308fdf6f73384958fac8d2dfc68"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.159161 27976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "c6f61309d827430c81aece8062159bfb"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "7c490df1fcb3267527913b2b3ef4aaf3"
server_key_iv: "180d3308fdf6f73384958fac8d2dfc68"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.160413 27976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:01:48.161136 27997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.161289 27976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:48.161373 27976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "c6f61309d827430c81aece8062159bfb"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "7c490df1fcb3267527913b2b3ef4aaf3"
server_key_iv: "180d3308fdf6f73384958fac8d2dfc68"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.161442 27976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:48.172763 27976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:48.173040 27976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:48.173164 27976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:48.173363 27976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:48.173679 27976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:48.173741 27976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.173789 27976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:48.173813 27976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.179365 27976 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:38287
I20260810 15:01:48.179430 28190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:38287 every 8 connection(s)
I20260810 15:01:48.179720 27976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:48.182813 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 27976
I20260810 15:01:48.182909 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:48.183079 29447 external_mini_cluster.cc:1428] Setting key 566327dbd6990c5f0dbb110114de80d9
I20260810 15:01:48.183959 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--builtin_ntp_servers=127.28.193.212:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:01:48.184729 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.180388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:45473 (local address 127.28.193.254:42775)
0810 15:01:48.180448 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.180450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.180679 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:01:48.181145 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.181146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.181154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.181209 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:48.181882 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.182367 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.183394 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.183600 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.183661 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.184469 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:48.184471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:48.184471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:48.184476 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:48.184492 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:48.184496 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:48.184552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.184617 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.184655 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:01:48.184865 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.180382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42775 (local address 127.28.193.193:45473)
0810 15:01:48.180561 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.180601 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.181063 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.181277 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.181281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.181289 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:48.181816 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.181822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.182430 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.182431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.183323 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.183326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.183681 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.183682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.183819 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.184336 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:48.184336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:48.184364 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:48.184387 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:48.184558 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:48.184560 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:48.184562 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.184605 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.184679 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260810 15:01:48.185820 28191 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42775
I20260810 15:01:48.185904 28191 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:48.186064 28191 heartbeater.cc:511] Master 127.28.193.254:42775 requested a full tablet report, sending...
I20260810 15:01:48.186605 27884 ts_manager.cc:194] Registered new tserver with Master: c6f61309d827430c81aece8062159bfb (127.28.193.193:38287)
I20260810 15:01:48.187407 27884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.193:45473
W20260810 15:01:48.265452 28195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:48.265655 28195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:48.265676 28195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:48.266798 28195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:48.266837 28195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:48.266860 28195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:48.268116 28195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:48.268301 28195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:48.268545 28195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:48.273483 28204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.273525 28207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.273469 28205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:48.273645 28195 server_base.cc:1034] running on GCE node
I20260810 15:01:48.273850 28195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:48.274057 28195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:48.275221 28195 hybrid_clock.cc:648] HybridClock initialized: now 1786374108275206 us; error 28 us; skew 500 ppm
I20260810 15:01:48.276398 28195 webserver.cc:460] Webserver started at http://127.28.193.194:33283/ using document root <none> and password file <none>
I20260810 15:01:48.276605 28195 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:48.276654 28195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:48.276791 28195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:48.277864 28195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "037866f011354d348d8e62b71cb58874"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "8cbfda452eb0fadbb4701b919fec34e8"
server_key_iv: "be392cfe0e2aff71437c14518892c750"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.278192 28195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "037866f011354d348d8e62b71cb58874"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "8cbfda452eb0fadbb4701b919fec34e8"
server_key_iv: "be392cfe0e2aff71437c14518892c750"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.279424 28195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:48.280185 28221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.280356 28195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:01:48.280449 28195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "037866f011354d348d8e62b71cb58874"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "8cbfda452eb0fadbb4701b919fec34e8"
server_key_iv: "be392cfe0e2aff71437c14518892c750"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.280521 28195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:48.297991 28195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:48.298280 28195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:48.298401 28195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:48.298625 28195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:48.298930 28195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:48.298964 28195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.298990 28195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:48.299014 28195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.304306 28195 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:36013
I20260810 15:01:48.304355 28412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:36013 every 8 connection(s)
I20260810 15:01:48.304646 28195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:01:48.309675 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.305385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:47517 (local address 127.28.193.254:42775)
0810 15:01:48.305444 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.305445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.305674 (+ 229us) server_negotiation.cc:407] Connection header received
0810 15:01:48.306254 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.306255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.306269 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.306327 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:48.306992 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.307488 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.308420 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.308608 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.308665 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.309417 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:48.309418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:48.309418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:48.309423 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:48.309440 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:48.309443 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:48.309487 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.309563 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.309595 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:01:48.309945 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.305381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42775 (local address 127.28.193.194:47517)
0810 15:01:48.305555 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.305588 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.306153 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.306398 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.306402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.306413 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:48.306897 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.306902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.307569 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.307570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.308333 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.308335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.308676 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.308677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.308806 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.309285 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:48.309286 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:48.309314 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:48.309337 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:48.309492 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:48.309497 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:48.309500 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.309559 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.309659 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260810 15:01:48.309939 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 28195
I20260810 15:01:48.310036 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:48.310237 29447 external_mini_cluster.cc:1428] Setting key a695f06f049ad0f19e5a31bbb5c61ec2
I20260810 15:01:48.310460 28413 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42775
I20260810 15:01:48.310557 28413 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:48.310734 28413 heartbeater.cc:511] Master 127.28.193.254:42775 requested a full tablet report, sending...
I20260810 15:01:48.311126 27884 ts_manager.cc:194] Registered new tserver with Master: 037866f011354d348d8e62b71cb58874 (127.28.193.194:36013)
I20260810 15:01:48.311169 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--builtin_ntp_servers=127.28.193.212:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:01:48.311501 27884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.194:47517
W20260810 15:01:48.392521 28418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:48.392719 28418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:48.392738 28418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:48.393855 28418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:48.393894 28418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:48.393918 28418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:48.395203 28418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:48.395393 28418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:48.395639 28418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:48.400456 28426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.400537 28425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:48.400421 28428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:48.400642 28418 server_base.cc:1034] running on GCE node
I20260810 15:01:48.400835 28418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:48.401083 28418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:48.402263 28418 hybrid_clock.cc:648] HybridClock initialized: now 1786374108402245 us; error 32 us; skew 500 ppm
I20260810 15:01:48.403507 28418 webserver.cc:460] Webserver started at http://127.28.193.195:34993/ using document root <none> and password file <none>
I20260810 15:01:48.403723 28418 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:48.403801 28418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:48.403931 28418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:48.404964 28418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "cd3e0717eba8411f9389f0b26584fe79"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "fcf25cabd2372b2dad022deaa6b4b01c"
server_key_iv: "b32c02f42a19b900672b14aeb3c757a6"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.405314 28418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "cd3e0717eba8411f9389f0b26584fe79"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "fcf25cabd2372b2dad022deaa6b4b01c"
server_key_iv: "b32c02f42a19b900672b14aeb3c757a6"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.406610 28418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:48.407254 28440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.407401 28418 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:48.407480 28418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "cd3e0717eba8411f9389f0b26584fe79"
format_stamp: "Formatted at 2026-08-10 15:01:48 on dist-test-slave-cvwc"
server_key: "fcf25cabd2372b2dad022deaa6b4b01c"
server_key_iv: "b32c02f42a19b900672b14aeb3c757a6"
server_key_version: "encryptionkey@0"
I20260810 15:01:48.407541 28418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:48.426155 28418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:48.426424 28418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:48.426548 28418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:48.426769 28418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:48.427093 28418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:48.427136 28418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.427165 28418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:48.427186 28418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:48.432595 28418 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:44249
I20260810 15:01:48.432632 28615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:44249 every 8 connection(s)
I20260810 15:01:48.432969 28418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:48.436754 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 28418
I20260810 15:01:48.436861 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:48.437042 29447 external_mini_cluster.cc:1428] Setting key d6d87681f81d0107872807c08c9e9a36
I20260810 15:01:48.438023 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.433669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:56033 (local address 127.28.193.254:42775)
0810 15:01:48.433741 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.433743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.433952 (+ 209us) server_negotiation.cc:407] Connection header received
0810 15:01:48.434452 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.434453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.434465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.434517 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:48.435161 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.435645 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.436606 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.436797 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.436859 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.437759 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:48.437761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:48.437761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:48.437766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:48.437782 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:48.437802 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:48.437841 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.437917 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.437944 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:01:48.438215 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.433652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:42775 (local address 127.28.193.195:56033)
0810 15:01:48.433839 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.433871 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.434357 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.434590 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.434594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.434604 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:48.435074 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.435079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.435730 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.435731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.436504 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.436506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.436892 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.436893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.437060 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.437591 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:48.437591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:48.437630 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:48.437657 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:48.437848 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:48.437854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:48.437857 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.437920 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.438022 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260810 15:01:48.438701 28618 heartbeater.cc:347] Connected to a master server at 127.28.193.254:42775
I20260810 15:01:48.438776 28618 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:48.438915 28618 heartbeater.cc:511] Master 127.28.193.254:42775 requested a full tablet report, sending...
I20260810 15:01:48.439222 27884 ts_manager.cc:194] Registered new tserver with Master: cd3e0717eba8411f9389f0b26584fe79 (127.28.193.195:44249)
I20260810 15:01:48.439582 27884 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.195:56033
I20260810 15:01:48.448698 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:48.452448 27946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.449967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.28.193.254:42775)
0810 15:01:48.450005 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.450007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.450100 (+ 93us) server_negotiation.cc:407] Connection header received
0810 15:01:48.450140 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.450141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.450149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.450197 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:48.450323 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.450781 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.451486 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.451655 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.451697 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.452273 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:48.452273 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:48.452274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:48.452277 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:48.452287 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:48.452290 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:48.452347 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.452387 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.452396 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:01:48.453912 27884 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:48.455785 27884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:48.461908 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.457832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38287 (local address 127.0.0.1:59024)
0810 15:01:48.458126 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.458168 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.458244 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.458630 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.458631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.458633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:48.458678 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.458680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459920 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.459921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.460686 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.460689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461561 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.461640 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461656 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.461755 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.461826 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":56,"threads_started":1}
I20260810 15:01:48.462029 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.457994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.28.193.195:44249)
0810 15:01:48.458262 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.458266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.458276 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:01:48.458326 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.458328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.458342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.458412 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:48.458585 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459667 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.460993 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461673 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461848 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.461911 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.461946 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260810 15:01:48.462169 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.458107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.28.193.194:36013)
0810 15:01:48.458327 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.458332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.458390 (+ 58us) server_negotiation.cc:407] Connection header received
0810 15:01:48.458461 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.458474 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.458489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.458772 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:48.458851 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459532 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461113 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461642 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461739 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461828 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.462059 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.462097 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260810 15:01:48.462013 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.457838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:44249 (local address 127.0.0.1:39494)
0810 15:01:48.458170 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.458180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.458245 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.458421 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.458424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.458426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:48.458503 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.458505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459745 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.459746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.460912 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.460916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461768 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.461839 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461843 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.461897 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.461939 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260810 15:01:48.462318 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.457838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36013 (local address 127.0.0.1:55652)
0810 15:01:48.458344 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.458352 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.458394 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.458692 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.458693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.458694 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:48.458723 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.458724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459616 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.459617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.461044 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.461046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461862 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.461919 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461922 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.461969 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.462008 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":40,"thread_start_us":112,"threads_started":1}
I20260810 15:01:48.462761 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.458007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.28.193.193:38287)
0810 15:01:48.458187 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.458192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.458493 (+ 301us) server_negotiation.cc:407] Connection header received
0810 15:01:48.458538 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.458540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.458555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.458631 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:48.458825 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.459851 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.460777 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.461472 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.461551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.461643 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.462639 (+ 996us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.462685 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260810 15:01:48.463043 28517 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 (DEFAULT_TABLE table=test-table [id=4c88c77204fa4a0690556d9daf119071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:48.463111 28308 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 (DEFAULT_TABLE table=test-table [id=4c88c77204fa4a0690556d9daf119071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:48.463439 28308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1b807a0a0e44269c97c1b92067ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:48.463443 28517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1b807a0a0e44269c97c1b92067ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:48.466089 28641 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Bootstrap starting.
I20260810 15:01:48.466395 28646 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Bootstrap starting.
I20260810 15:01:48.466753 28641 tablet_bootstrap.cc:654] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:48.466681 28074 tablet_service.cc:1467] Processing CreateTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 (DEFAULT_TABLE table=test-table [id=4c88c77204fa4a0690556d9daf119071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:48.466956 28646 tablet_bootstrap.cc:654] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:48.467023 28074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f1b807a0a0e44269c97c1b92067ede1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:48.467077 28641 log.cc:824] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:48.467238 28646 log.cc:824] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:48.467867 28641 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: No bootstrap required, opened a new log
I20260810 15:01:48.467924 28646 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: No bootstrap required, opened a new log
I20260810 15:01:48.467942 28641 ts_tablet_manager.cc:1397] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:48.467970 28646 ts_tablet_manager.cc:1397] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:48.469579 28651 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: Bootstrap starting.
I20260810 15:01:48.470245 28651 tablet_bootstrap.cc:654] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:48.470517 28651 log.cc:824] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:48.470428 28641 raft_consensus.cc:348] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.470588 28641 raft_consensus.cc:374] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:48.470630 28641 raft_consensus.cc:729] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3e0717eba8411f9389f0b26584fe79, State: Initialized, Role: FOLLOWER
I20260810 15:01:48.470762 28641 consensus_queue.cc:260] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.470783 28646 raft_consensus.cc:348] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.470928 28646 raft_consensus.cc:374] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:48.470965 28646 raft_consensus.cc:729] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037866f011354d348d8e62b71cb58874, State: Initialized, Role: FOLLOWER
I20260810 15:01:48.471043 28641 ts_tablet_manager.cc:1428] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:01:48.471083 28646 consensus_queue.cc:260] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.471163 28618 heartbeater.cc:503] Master 127.28.193.254:42775 was elected leader, sending a full tablet report...
I20260810 15:01:48.471186 28651 tablet_bootstrap.cc:492] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: No bootstrap required, opened a new log
I20260810 15:01:48.471251 28651 ts_tablet_manager.cc:1397] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:48.471335 28646 ts_tablet_manager.cc:1428] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:01:48.473165 28651 raft_consensus.cc:348] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.473286 28651 raft_consensus.cc:374] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:48.473317 28651 raft_consensus.cc:729] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f61309d827430c81aece8062159bfb, State: Initialized, Role: FOLLOWER
I20260810 15:01:48.473421 28651 consensus_queue.cc:260] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.473644 28651 ts_tablet_manager.cc:1428] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:01:48.473743 28191 heartbeater.cc:503] Master 127.28.193.254:42775 was elected leader, sending a full tablet report...
I20260810 15:01:48.473778 28413 heartbeater.cc:503] Master 127.28.193.254:42775 was elected leader, sending a full tablet report...
W20260810 15:01:48.555181 28414 tablet.cc:2369] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:48.565331 28653 raft_consensus.cc:482] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:48.565482 28653 raft_consensus.cc:504] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.565907 28653 leader_election.cc:288] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037866f011354d348d8e62b71cb58874 (127.28.193.194:36013), c6f61309d827430c81aece8062159bfb (127.28.193.193:38287)
I20260810 15:01:48.568781 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.566201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:53299 (local address 127.28.193.194:36013)
0810 15:01:48.566250 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.566252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.566279 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:01:48.566320 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.566322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.566332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.566384 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:48.566557 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.567067 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568052 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.568435 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568493 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.568502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.568662 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.568705 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:01:48.568790 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.566101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36013 (local address 127.28.193.195:53299)
0810 15:01:48.566158 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.566172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.566254 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.566399 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.566400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.566401 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:48.566483 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.566485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.567146 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.567147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.567938 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.567942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.568511 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.568587 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.568591 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.568645 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.568692 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:01:48.568784 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.566236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:60605 (local address 127.28.193.193:38287)
0810 15:01:48.566284 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.566285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.566373 (+ 88us) server_negotiation.cc:407] Connection header received
0810 15:01:48.566420 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.566421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.566433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.566500 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:48.566650 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.567135 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568076 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.568450 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568527 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.568536 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.568668 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.568704 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:01:48.568835 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.566101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38287 (local address 127.28.193.195:60605)
0810 15:01:48.566298 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.566307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:48.566370 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:48.566507 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:48.566507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:48.566509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:48.566578 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.566579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.567240 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.567241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.567973 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:48.567975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.568538 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.568539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:48.568596 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.568599 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:48.568647 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:01:48.568690 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":15,"thread_start_us":105,"threads_started":1}
I20260810 15:01:48.569145 28338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1b807a0a0e44269c97c1b92067ede1" candidate_uuid: "cd3e0717eba8411f9389f0b26584fe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037866f011354d348d8e62b71cb58874" is_pre_election: true
I20260810 15:01:48.569310 28338 raft_consensus.cc:2454] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3e0717eba8411f9389f0b26584fe79 in term 0.
I20260810 15:01:48.569339 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1b807a0a0e44269c97c1b92067ede1" candidate_uuid: "cd3e0717eba8411f9389f0b26584fe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f61309d827430c81aece8062159bfb" is_pre_election: true
I20260810 15:01:48.569473 28110 raft_consensus.cc:2454] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd3e0717eba8411f9389f0b26584fe79 in term 0.
I20260810 15:01:48.569555 28447 leader_election.cc:302] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037866f011354d348d8e62b71cb58874, cd3e0717eba8411f9389f0b26584fe79; no voters:
I20260810 15:01:48.569702 28653 raft_consensus.cc:2781] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:48.569798 28653 raft_consensus.cc:482] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:48.569826 28653 raft_consensus.cc:3037] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:48.570803 28653 raft_consensus.cc:504] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.570956 28653 leader_election.cc:288] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [CANDIDATE]: Term 1 election: Requested vote from peers 037866f011354d348d8e62b71cb58874 (127.28.193.194:36013), c6f61309d827430c81aece8062159bfb (127.28.193.193:38287)
I20260810 15:01:48.571138 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1b807a0a0e44269c97c1b92067ede1" candidate_uuid: "cd3e0717eba8411f9389f0b26584fe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f61309d827430c81aece8062159bfb"
I20260810 15:01:48.571224 28110 raft_consensus.cc:3037] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:48.572038 28110 raft_consensus.cc:2454] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3e0717eba8411f9389f0b26584fe79 in term 1.
I20260810 15:01:48.572167 28338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f1b807a0a0e44269c97c1b92067ede1" candidate_uuid: "cd3e0717eba8411f9389f0b26584fe79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037866f011354d348d8e62b71cb58874"
I20260810 15:01:48.572224 28441 leader_election.cc:302] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6f61309d827430c81aece8062159bfb, cd3e0717eba8411f9389f0b26584fe79; no voters:
I20260810 15:01:48.572252 28338 raft_consensus.cc:3037] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:48.572304 28653 raft_consensus.cc:2781] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:48.572438 28653 raft_consensus.cc:686] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 1 LEADER]: Becoming Leader. State: Replica: cd3e0717eba8411f9389f0b26584fe79, State: Running, Role: LEADER
I20260810 15:01:48.572527 28653 consensus_queue.cc:237] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } }
I20260810 15:01:48.573032 28338 raft_consensus.cc:2454] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd3e0717eba8411f9389f0b26584fe79 in term 1.
I20260810 15:01:48.573540 27883 catalog_manager.cc:5495] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd3e0717eba8411f9389f0b26584fe79 (127.28.193.195). New cstate: current_term: 1 leader_uuid: "cd3e0717eba8411f9389f0b26584fe79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3e0717eba8411f9389f0b26584fe79" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 44249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:48.607132 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.604856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.28.193.195:44249)
0810 15:01:48.604910 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:48.604912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:48.604921 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:01:48.604965 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:48.604966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:48.604986 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:48.605047 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:48.605156 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.605650 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.606416 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:48.606604 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:48.606657 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:48.606801 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:48.606891 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:48.606977 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:48.607024 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:01:48.607031 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:01:48.608963 28338 raft_consensus.cc:1261] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 1 FOLLOWER]: Refusing update from remote peer cd3e0717eba8411f9389f0b26584fe79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:48.608968 28110 raft_consensus.cc:1261] T 2f1b807a0a0e44269c97c1b92067ede1 P c6f61309d827430c81aece8062159bfb [term 1 FOLLOWER]: Refusing update from remote peer cd3e0717eba8411f9389f0b26584fe79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:48.609304 28659 consensus_queue.cc:1035] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037866f011354d348d8e62b71cb58874" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:48.609424 28653 consensus_queue.cc:1035] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f61309d827430c81aece8062159bfb" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 38287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:48.611907 28664 mvcc.cc:205] Tried to move back new op lower bound from 7316988348859699200 to 7316988348713750528. Current Snapshot: MvccSnapshot[applied={T|T < 7316988348859699200}]
I20260810 15:01:48.612084 28666 mvcc.cc:205] Tried to move back new op lower bound from 7316988348859699200 to 7316988348713750528. Current Snapshot: MvccSnapshot[applied={T|T < 7316988348859699200}]
I20260810 15:01:48.612323 28667 mvcc.cc:205] Tried to move back new op lower bound from 7316988348859699200 to 7316988348713750528. Current Snapshot: MvccSnapshot[applied={T|T < 7316988348859699200}]
I20260810 15:01:48.614610 27884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:48.614712 27884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:48.615443 27884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c9d5b448c974013ae3ca9169dcc5b86: Sending DeleteTablet for 3 replicas of tablet 2f1b807a0a0e44269c97c1b92067ede1
I20260810 15:01:48.615746 28517 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:48 UTC) from {username='slave'} at 127.0.0.1:39494
I20260810 15:01:48.615768 28308 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:48 UTC) from {username='slave'} at 127.0.0.1:55652
I20260810 15:01:48.615960 28074 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f1b807a0a0e44269c97c1b92067ede1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:48 UTC) from {username='slave'} at 127.0.0.1:59024
I20260810 15:01:48.616058 28679 tablet_replica.cc:330] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: stopping tablet replica
I20260810 15:01:48.616153 28679 raft_consensus.cc:2229] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:48.616256 28679 raft_consensus.cc:2258] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:48.616374 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 27976
I20260810 15:01:48.616420 28678 tablet_replica.cc:330] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: stopping tablet replica
I20260810 15:01:48.616499 28678 raft_consensus.cc:2229] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:01:48.635710 28678 raft_consensus.cc:2258] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:48.635829 28679 ts_tablet_manager.cc:1905] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:48.636284 28678 ts_tablet_manager.cc:1905] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:48.637585 28679 ts_tablet_manager.cc:1918] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:48.637676 28679 log.cc:1196] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/2f1b807a0a0e44269c97c1b92067ede1
I20260810 15:01:48.637961 28679 ts_tablet_manager.cc:1939] T 2f1b807a0a0e44269c97c1b92067ede1 P 037866f011354d348d8e62b71cb58874: Deleting consensus metadata
I20260810 15:01:48.638123 28678 ts_tablet_manager.cc:1918] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:48.638244 28678 log.cc:1196] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/2f1b807a0a0e44269c97c1b92067ede1
I20260810 15:01:48.638446 27864 catalog_manager.cc:4841] TS 037866f011354d348d8e62b71cb58874 (127.28.193.194:36013): tablet 2f1b807a0a0e44269c97c1b92067ede1 (table test-table [id=4c88c77204fa4a0690556d9daf119071]) successfully deleted
I20260810 15:01:48.638589 28678 ts_tablet_manager.cc:1939] T 2f1b807a0a0e44269c97c1b92067ede1 P cd3e0717eba8411f9389f0b26584fe79: Deleting consensus metadata
I20260810 15:01:48.638999 27861 catalog_manager.cc:4841] TS cd3e0717eba8411f9389f0b26584fe79 (127.28.193.195:44249): tablet 2f1b807a0a0e44269c97c1b92067ede1 (table test-table [id=4c88c77204fa4a0690556d9daf119071]) successfully deleted
W20260810 15:01:48.640125 27860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:38287 (error 108)
I20260810 15:01:48.640343 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 28195
I20260810 15:01:48.642323 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:48.640494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:38287 (local address 127.0.0.1:59038)
0810 15:01:48.642243 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:48.642291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1723}
W20260810 15:01:48.644495 27860 catalog_manager.cc:4584] TS c6f61309d827430c81aece8062159bfb (127.28.193.193:38287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f1b807a0a0e44269c97c1b92067ede1: Network error: Client connection negotiation failed: client connection to 127.28.193.193:38287: connect: Connection refused (error 111)
I20260810 15:01:48.658595 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 28418
I20260810 15:01:48.669770 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 27833
2026-08-10T15:01:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:48 dist-test-slave-cvwc krb5kdc[28689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:48 dist-test-slave-cvwc krb5kdc[28689](info): set up 2 sockets
Aug 10 15:01:48 dist-test-slave-cvwc krb5kdc[28689](info): commencing operation
krb5kdc: starting...
W20260810 15:01:49.850246 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:49 dist-test-slave-cvwc krb5kdc[28689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:01:49.968608 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34341
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:50.051352 28708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:50.051568 28708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:50.051592 28708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:50.052772 28708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:50.052819 28708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:50.052834 28708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:50.052847 28708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:50.052860 28708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:50.054373 28708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:50.054612 28708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:50.054847 28708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:50.059679 28721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.059685 28713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.059733 28715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:50.059921 28708 server_base.cc:1034] running on GCE node
I20260810 15:01:50.060214 28708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:50.060487 28708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:50.061638 28708 hybrid_clock.cc:648] HybridClock initialized: now 1786374110061624 us; error 32 us; skew 500 ppm
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:50.064179 28708 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:01:50.064584 28708 webserver.cc:460] Webserver started at http://127.28.193.254:33595/ using document root <none> and password file <none>
I20260810 15:01:50.064783 28708 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:50.064831 28708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:50.064965 28708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:50.066079 28708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "e160815a405641399035e5b09a769311"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "323cb93a40ffdf146684dcf12ceb0903"
server_key_iv: "a554da56b8ae706965797a6aa5d7c7c3"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.066421 28708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "e160815a405641399035e5b09a769311"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "323cb93a40ffdf146684dcf12ceb0903"
server_key_iv: "a554da56b8ae706965797a6aa5d7c7c3"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.067687 28708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:01:50.068394 28733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.068569 28708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:50.068653 28708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "e160815a405641399035e5b09a769311"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "323cb93a40ffdf146684dcf12ceb0903"
server_key_iv: "a554da56b8ae706965797a6aa5d7c7c3"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.068714 28708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:50.088315 28708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:50.090453 28708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:50.090592 28708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:50.094250 28708 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34341
I20260810 15:01:50.094290 28824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34341 every 8 connection(s)
I20260810 15:01:50.094612 28708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:50.095522 28825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:50.098021 28825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: Bootstrap starting.
I20260810 15:01:50.098598 28825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:50.098863 28825 log.cc:824] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:50.099522 28825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: No bootstrap required, opened a new log
I20260810 15:01:50.101047 28825 raft_consensus.cc:348] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } }
I20260810 15:01:50.101186 28825 raft_consensus.cc:374] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:50.101218 28825 raft_consensus.cc:729] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e160815a405641399035e5b09a769311, State: Initialized, Role: FOLLOWER
I20260810 15:01:50.101356 28825 consensus_queue.cc:260] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } }
I20260810 15:01:50.101438 28825 raft_consensus.cc:388] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:50.101470 28825 raft_consensus.cc:482] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:50.101517 28825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:50.102046 28825 raft_consensus.cc:504] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } }
I20260810 15:01:50.102144 28825 leader_election.cc:302] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e160815a405641399035e5b09a769311; no voters:
I20260810 15:01:50.102308 28825 leader_election.cc:288] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:50.102334 28830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:50.102483 28830 raft_consensus.cc:686] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [term 1 LEADER]: Becoming Leader. State: Replica: e160815a405641399035e5b09a769311, State: Running, Role: LEADER
I20260810 15:01:50.102551 28825 sys_catalog.cc:564] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:50.102573 28830 consensus_queue.cc:237] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } }
I20260810 15:01:50.102928 28832 sys_catalog.cc:455] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e160815a405641399035e5b09a769311. Latest consensus state: current_term: 1 leader_uuid: "e160815a405641399035e5b09a769311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } } }
I20260810 15:01:50.103010 28832 sys_catalog.cc:458] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:50.103142 28831 sys_catalog.cc:455] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e160815a405641399035e5b09a769311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e160815a405641399035e5b09a769311" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34341 } } }
I20260810 15:01:50.103231 28845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:50.103255 28831 sys_catalog.cc:458] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:50.103782 28845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:50.103894 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 28708
I20260810 15:01:50.104019 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:50.104204 29447 external_mini_cluster.cc:1428] Setting key 181693106ad5f53e4caef6db06c12329
I20260810 15:01:50.105278 28845 catalog_manager.cc:1347] Generated new cluster ID: e2470207035945d0ada48c15082c511b
I20260810 15:01:50.105333 28845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:50.115690 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.104722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49172 (local address 127.28.193.254:34341)
0810 15:01:50.104889 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.104900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.104934 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:01:50.105059 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.105067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.105122 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.105281 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:50.106125 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.107631 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.108651 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.109268 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.109390 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.112154 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:50.112159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:50.112159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:50.112172 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:50.114075 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.114473 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.114476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.114477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.114579 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.114755 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.114756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.114756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.115067 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:50.115213 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.115323 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.115459 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260810 15:01:50.123481 28845 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:50.124251 28845 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:50.129081 28845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: Generated new TSK 0
I20260810 15:01:50.129263 28845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.193.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
Entry for principal kudu/127.28.193.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab.
I20260810 15:01:50.155815 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:50.238191 28868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:50.238401 28868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:50.238423 28868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:50.239586 28868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:50.239627 28868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:50.239652 28868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:50.241151 28868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.193.keytab
--principal=kudu/127.28.193.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:50.241382 28868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:50.241605 28868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:50.246414 28876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.246411 28877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.246462 28879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:50.246656 28868 server_base.cc:1034] running on GCE node
I20260810 15:01:50.246845 28868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:50.247043 28868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:50.248212 28868 hybrid_clock.cc:648] HybridClock initialized: now 1786374110248189 us; error 34 us; skew 500 ppm
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:50.253809 28868 init.cc:377] Logged in from keytab as kudu/127.28.193.193@KRBTEST.COM (short username kudu)
I20260810 15:01:50.254261 28868 webserver.cc:460] Webserver started at http://127.28.193.193:35023/ using document root <none> and password file <none>
I20260810 15:01:50.254487 28868 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:50.254531 28868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:50.254648 28868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:50.255692 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "2a88e0e49b8a4091a3105d6fe7ace519"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "22ab5e9675bc2b2f9f328453f9d18cd6"
server_key_iv: "c91ead74974bf98abed76378cdde7ef1"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.256023 28868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "2a88e0e49b8a4091a3105d6fe7ace519"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "22ab5e9675bc2b2f9f328453f9d18cd6"
server_key_iv: "c91ead74974bf98abed76378cdde7ef1"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.257373 28868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:50.258142 28889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.258306 28868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.258397 28868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "2a88e0e49b8a4091a3105d6fe7ace519"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "22ab5e9675bc2b2f9f328453f9d18cd6"
server_key_iv: "c91ead74974bf98abed76378cdde7ef1"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.258486 28868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:50.286909 28868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:50.289433 28868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:50.289588 28868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:50.289839 28868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:50.290189 28868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:50.290225 28868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.290256 28868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:50.290275 28868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.301632 28868 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:42065
I20260810 15:01:50.301676 29084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:42065 every 8 connection(s)
I20260810 15:01:50.302120 28868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:50.311408 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 28868
I20260810 15:01:50.311528 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:50.311776 29447 external_mini_cluster.cc:1428] Setting key 088174bc5f960105b518ae79d3fba6fc
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.193@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:50.316361 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.302904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:52409 (local address 127.28.193.254:34341)
0810 15:01:50.302974 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.302976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.303243 (+ 267us) server_negotiation.cc:407] Connection header received
0810 15:01:50.304002 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.304004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.304019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.304091 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:50.305334 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.306162 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.307407 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.307656 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.307689 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.313020 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:50.313022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:50.313022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:50.313031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:50.314639 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.315092 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.315094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.315094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.315166 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.315504 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.315505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.315505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.315650 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:50.315697 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.316139 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.316257 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:01:50.316962 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.302909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34341 (local address 127.28.193.193:52409)
0810 15:01:50.303096 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.303150 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.303826 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.304174 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.304179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.304934 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:50.305223 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.305231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.306265 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.306266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.307303 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.307307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.307769 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.307769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.307893 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.311027 (+ 3134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:50.311039 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:50.312912 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:50.314712 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.314721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.314724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.315024 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.315261 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.315262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.315263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.315415 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.315746 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:50.315754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:50.315964 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.316506 (+ 542us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.316703 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260810 15:01:50.317500 29085 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34341
I20260810 15:01:50.317613 29085 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:50.317858 29085 heartbeater.cc:511] Master 127.28.193.254:34341 requested a full tablet report, sending...
I20260810 15:01:50.318491 28762 ts_manager.cc:194] Registered new tserver with Master: 2a88e0e49b8a4091a3105d6fe7ace519 (127.28.193.193:42065)
I20260810 15:01:50.319392 28762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.193@KRBTEST.COM'} at 127.28.193.193:52409
WARNING: no policy specified for kudu/127.28.193.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
Entry for principal kudu/127.28.193.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab.
I20260810 15:01:50.338798 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:50.420528 29096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:50.420722 29096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:50.420756 29096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:50.421903 29096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:50.421943 29096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:50.421967 29096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:50.423388 29096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.194.keytab
--principal=kudu/127.28.193.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:50.423596 29096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:50.423810 29096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:50.428637 29103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.428643 29106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.428648 29104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:50.428926 29096 server_base.cc:1034] running on GCE node
I20260810 15:01:50.429123 29096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:50.429337 29096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:50.430485 29096 hybrid_clock.cc:648] HybridClock initialized: now 1786374110430471 us; error 26 us; skew 500 ppm
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:50.432864 29096 init.cc:377] Logged in from keytab as kudu/127.28.193.194@KRBTEST.COM (short username kudu)
I20260810 15:01:50.433271 29096 webserver.cc:460] Webserver started at http://127.28.193.194:38335/ using document root <none> and password file <none>
I20260810 15:01:50.433445 29096 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:50.433482 29096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:50.433571 29096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:50.434614 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "74c2960510514dc2987d21f40bccdbf7"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "d9ed3a822dbcd8b737df6529d693be82"
server_key_iv: "e629919a12e3b0a67b6fa0fea4b325ae"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.434921 29096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "74c2960510514dc2987d21f40bccdbf7"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "d9ed3a822dbcd8b737df6529d693be82"
server_key_iv: "e629919a12e3b0a67b6fa0fea4b325ae"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.436074 29096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:01:50.436779 29121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.436929 29096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:50.437021 29096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "74c2960510514dc2987d21f40bccdbf7"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "d9ed3a822dbcd8b737df6529d693be82"
server_key_iv: "e629919a12e3b0a67b6fa0fea4b325ae"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.437093 29096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:50.459553 29096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:50.461335 29096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:50.461464 29096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:50.461691 29096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:50.462044 29096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:50.462083 29096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.462105 29096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:50.462137 29096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.467620 29096 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:38331
I20260810 15:01:50.467665 29316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:38331 every 8 connection(s)
I20260810 15:01:50.467952 29096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.194@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:50.473688 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29096
I20260810 15:01:50.473816 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:50.474025 29447 external_mini_cluster.cc:1428] Setting key f3c710a80796f29d1df54f03fcb994a8
I20260810 15:01:50.476819 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.468632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:38361 (local address 127.28.193.254:34341)
0810 15:01:50.468691 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.468693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.468918 (+ 225us) server_negotiation.cc:407] Connection header received
0810 15:01:50.469374 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.469375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.469387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.469446 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:50.470400 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.470952 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.471915 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.472108 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.472171 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.474239 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:50.474241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:50.474242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:50.474250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:50.475503 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.475934 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.475935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.475935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.475981 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.476219 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.476220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.476220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.476333 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:50.476390 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.476651 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.476734 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:01:50.476989 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.468600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34341 (local address 127.28.193.194:38361)
0810 15:01:50.468755 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.468803 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.469286 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.469519 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.469523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.470120 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:50.470315 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.470320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.471017 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.471018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.471833 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.471836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.472168 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.472169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.472261 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.472679 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:50.472688 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:50.474157 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:50.475577 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.475585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.475587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.475868 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.476031 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.476032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.476032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.476159 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.476396 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:50.476404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:50.476532 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.476621 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.476765 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260810 15:01:50.477371 29317 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34341
I20260810 15:01:50.477479 29317 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:50.477639 29317 heartbeater.cc:511] Master 127.28.193.254:34341 requested a full tablet report, sending...
I20260810 15:01:50.478010 28762 ts_manager.cc:194] Registered new tserver with Master: 74c2960510514dc2987d21f40bccdbf7 (127.28.193.194:38331)
I20260810 15:01:50.478358 28762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.194@KRBTEST.COM'} at 127.28.193.194:38361
WARNING: no policy specified for kudu/127.28.193.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
Entry for principal kudu/127.28.193.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab.
I20260810 15:01:50.497828 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:50.578867 29327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:50.579068 29327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:50.579090 29327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:50.580210 29327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:50.580253 29327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:50.580284 29327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:50.581765 29327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.195.keytab
--principal=kudu/127.28.193.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:50.581996 29327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:50.582238 29327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:01:50.587188 29327 server_base.cc:1034] running on GCE node
W20260810 15:01:50.587146 29340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.587164 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:50.587157 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:50.587570 29327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:50.587798 29327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:50.588961 29327 hybrid_clock.cc:648] HybridClock initialized: now 1786374110588948 us; error 32 us; skew 500 ppm
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:50.591495 29327 init.cc:377] Logged in from keytab as kudu/127.28.193.195@KRBTEST.COM (short username kudu)
I20260810 15:01:50.591907 29327 webserver.cc:460] Webserver started at http://127.28.193.195:45841/ using document root <none> and password file <none>
I20260810 15:01:50.592125 29327 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:50.592173 29327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:50.592298 29327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:50.593276 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "c8029f68ab0b488dbb787be3be96ae3e"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "c8002133f1f00fb8b36e2321a1fc85cb"
server_key_iv: "3e0196b1f682ce592a1d6761dc8bed1d"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.593606 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "c8029f68ab0b488dbb787be3be96ae3e"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "c8002133f1f00fb8b36e2321a1fc85cb"
server_key_iv: "3e0196b1f682ce592a1d6761dc8bed1d"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.594868 29327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:50.595535 29356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.595685 29327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:50.595765 29327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "c8029f68ab0b488dbb787be3be96ae3e"
format_stamp: "Formatted at 2026-08-10 15:01:50 on dist-test-slave-cvwc"
server_key: "c8002133f1f00fb8b36e2321a1fc85cb"
server_key_iv: "3e0196b1f682ce592a1d6761dc8bed1d"
server_key_version: "encryptionkey@0"
I20260810 15:01:50.595840 29327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:50.610121 29327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:50.611930 29327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:50.612056 29327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:50.612255 29327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:50.612540 29327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:50.612572 29327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.612599 29327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:50.612627 29327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:50.618225 29327 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:45121
I20260810 15:01:50.618306 29555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:45121 every 8 connection(s)
I20260810 15:01:50.618652 29327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:50.622677 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29327
I20260810 15:01:50.622774 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:50.622956 29447 external_mini_cluster.cc:1428] Setting key e22a0b19dbda25929944090b8bd6afe1
Aug 10 15:01:50 dist-test-slave-cvwc krb5kdc[28689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374110, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.195@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:01:50.627383 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.619367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:42497 (local address 127.28.193.254:34341)
0810 15:01:50.619439 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.619441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.619601 (+ 160us) server_negotiation.cc:407] Connection header received
0810 15:01:50.620070 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.620072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.620083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.620143 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:50.621040 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.621536 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.622490 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.622686 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.622738 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.624955 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:50.624957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:50.624957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:50.624965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:50.626223 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.626561 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.626562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.626562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.626608 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.626854 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.626854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.626854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.626949 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:50.626995 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.627261 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.627322 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:01:50.627529 29563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.619322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34341 (local address 127.28.193.195:42497)
0810 15:01:50.619481 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.619520 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.619982 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.620204 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.620207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.620755 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:50.620940 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.620944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.621605 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.621605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.622428 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.622430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.622762 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.622763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.622904 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.623415 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:01:50.623425 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:50.624857 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:50.626291 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.626297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.626299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.626504 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.626677 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:01:50.626678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:01:50.626678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:01:50.626797 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:01:50.627015 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:50.627021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:50.627147 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.627234 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.627349 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260810 15:01:50.627923 29557 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34341
I20260810 15:01:50.628003 29557 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:50.628144 29557 heartbeater.cc:511] Master 127.28.193.254:34341 requested a full tablet report, sending...
I20260810 15:01:50.628480 28762 ts_manager.cc:194] Registered new tserver with Master: c8029f68ab0b488dbb787be3be96ae3e (127.28.193.195:45121)
I20260810 15:01:50.628836 28762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.193.195@KRBTEST.COM'} at 127.28.193.195:42497
I20260810 15:01:50.634496 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:50.641158 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.635718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.28.193.254:34341)
0810 15:01:50.635789 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.635790 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.635862 (+ 72us) server_negotiation.cc:407] Connection header received
0810 15:01:50.635930 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.635931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.635940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.635980 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:50.636485 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.636945 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.637618 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.637831 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.637880 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.638988 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:50.638990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:50.638990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:50.638997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:50.640129 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.640445 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.640446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.640446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.640499 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:50.640663 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:50.640663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:50.640664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:50.640816 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:50.640915 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.641003 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.641074 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:01:50.642484 28762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:50.644137 28762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:50.650341 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.28.193.193:42065)
0810 15:01:50.646815 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.646819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.646827 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:01:50.646858 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.646860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.646871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.646948 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:50.647119 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.647922 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.649379 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.649961 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650030 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.650088 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.650226 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.650265 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":85,"threads_started":1}
I20260810 15:01:50.650326 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:42065 (local address 127.0.0.1:39152)
0810 15:01:50.646441 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.646483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.646549 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.646972 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.646975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.646977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:50.647041 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.647043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.648011 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.648012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.649289 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.649292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.650035 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.650116 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.650130 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.650192 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.650247 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":41,"thread_start_us":47,"threads_started":1}
I20260810 15:01:50.650687 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:38331 (local address 127.0.0.1:57054)
0810 15:01:50.646557 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.646567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.646619 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.647303 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.647304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.647306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:50.647352 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.647353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.648503 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.648504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.649332 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.649334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.650316 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.650382 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.650524 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.650573 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.650617 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260810 15:01:50.650727 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.28.193.194:38331)
0810 15:01:50.647081 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.647085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.647094 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:01:50.647130 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.647131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.647146 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.647218 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:50.647428 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.648417 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.649422 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.650162 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650231 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.650280 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.650603 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.650648 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":87,"threads_started":1}
I20260810 15:01:50.651100 28974 tablet_service.cc:1467] Processing CreateTablet for tablet 3313145c48fe477eb64cd6cff4d1298b (DEFAULT_TABLE table=test-table [id=40e55904374d4940865922dcf6822ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:50.651463 28974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3313145c48fe477eb64cd6cff4d1298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:50.651410 29207 tablet_service.cc:1467] Processing CreateTablet for tablet 3313145c48fe477eb64cd6cff4d1298b (DEFAULT_TABLE table=test-table [id=40e55904374d4940865922dcf6822ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:50.651722 29207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3313145c48fe477eb64cd6cff4d1298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:50.654055 29590 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Bootstrap starting.
I20260810 15:01:50.654290 29591 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Bootstrap starting.
I20260810 15:01:50.654428 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:45121 (local address 127.0.0.1:45966)
0810 15:01:50.646677 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.646685 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.646721 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.647035 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.647036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.647038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:50.647066 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.647067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.648277 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.648278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.649342 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.649345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.650318 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.650382 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.654258 (+ 3876us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.654309 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.654357 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":3867,"thread_start_us":74,"threads_started":1}
I20260810 15:01:50.654758 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.646675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.28.193.195:45121)
0810 15:01:50.646827 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.646832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.646841 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:01:50.646878 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.646880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.646892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.646964 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:50.647138 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.648174 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.649424 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.650162 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.650231 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.650280 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.654606 (+ 4326us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.654653 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260810 15:01:50.654899 29590 tablet_bootstrap.cc:654] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:50.654937 29591 tablet_bootstrap.cc:654] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:50.655376 29591 log.cc:824] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:50.655478 29590 log.cc:824] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:50.655390 29441 tablet_service.cc:1467] Processing CreateTablet for tablet 3313145c48fe477eb64cd6cff4d1298b (DEFAULT_TABLE table=test-table [id=40e55904374d4940865922dcf6822ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:50.655707 29441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3313145c48fe477eb64cd6cff4d1298b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:50.656083 29591 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: No bootstrap required, opened a new log
I20260810 15:01:50.656143 29591 ts_tablet_manager.cc:1397] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:50.657549 29591 raft_consensus.cc:348] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.657749 29591 raft_consensus.cc:374] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:50.657801 29591 raft_consensus.cc:729] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c2960510514dc2987d21f40bccdbf7, State: Initialized, Role: FOLLOWER
I20260810 15:01:50.657913 29591 consensus_queue.cc:260] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.658215 29591 ts_tablet_manager.cc:1428] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:01:50.658334 29590 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: No bootstrap required, opened a new log
I20260810 15:01:50.658399 29317 heartbeater.cc:503] Master 127.28.193.254:34341 was elected leader, sending a full tablet report...
I20260810 15:01:50.658421 29590 ts_tablet_manager.cc:1397] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 15:01:50.658515 29599 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Bootstrap starting.
I20260810 15:01:50.659075 29599 tablet_bootstrap.cc:654] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:50.659576 29599 log.cc:824] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:50.660202 29599 tablet_bootstrap.cc:492] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: No bootstrap required, opened a new log
I20260810 15:01:50.660130 29590 raft_consensus.cc:348] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.660256 29599 ts_tablet_manager.cc:1397] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:50.660264 29590 raft_consensus.cc:374] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:50.660310 29590 raft_consensus.cc:729] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a88e0e49b8a4091a3105d6fe7ace519, State: Initialized, Role: FOLLOWER
I20260810 15:01:50.660427 29590 consensus_queue.cc:260] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.660696 29590 ts_tablet_manager.cc:1428] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:01:50.660734 29085 heartbeater.cc:503] Master 127.28.193.254:34341 was elected leader, sending a full tablet report...
I20260810 15:01:50.661664 29599 raft_consensus.cc:348] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.661839 29599 raft_consensus.cc:374] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:50.661886 29599 raft_consensus.cc:729] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8029f68ab0b488dbb787be3be96ae3e, State: Initialized, Role: FOLLOWER
I20260810 15:01:50.662009 29599 consensus_queue.cc:260] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.662253 29599 ts_tablet_manager.cc:1428] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:50.662307 29557 heartbeater.cc:503] Master 127.28.193.254:34341 was elected leader, sending a full tablet report...
W20260810 15:01:50.718556 29319 tablet.cc:2369] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:01:50.802922 29088 tablet.cc:2369] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:01:50.817119 29602 raft_consensus.cc:482] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:50.817245 29602 raft_consensus.cc:504] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.817624 29602 leader_election.cc:288] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8029f68ab0b488dbb787be3be96ae3e (127.28.193.195:45121), 74c2960510514dc2987d21f40bccdbf7 (127.28.193.194:38331)
I20260810 15:01:50.820489 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.817987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:38671 (local address 127.28.193.195:45121)
0810 15:01:50.818042 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.818044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.818090 (+ 46us) server_negotiation.cc:407] Connection header received
0810 15:01:50.818121 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.818123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.818132 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.818182 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:50.818348 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.818836 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.819801 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.820167 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.820227 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.820237 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.820385 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.820421 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:01:50.820489 29585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.817940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:53971 (local address 127.28.193.194:38331)
0810 15:01:50.817994 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.817996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.818075 (+ 79us) server_negotiation.cc:407] Connection header received
0810 15:01:50.818162 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.818164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.818175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.818228 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:50.818371 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.818847 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.819802 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.820169 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.820235 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.820243 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.820382 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.820418 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:01:50.820538 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.817850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:38331 (local address 127.28.193.193:53971)
0810 15:01:50.818009 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.818020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.818084 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.818241 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.818242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.818244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:50.818307 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.818309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.818928 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.818929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.819709 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.819711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.820248 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.820249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.820316 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.820320 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.820369 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.820411 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260810 15:01:50.820541 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.817847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:45121 (local address 127.28.193.193:38671)
0810 15:01:50.818009 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.818021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:50.818077 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:50.818187 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:50.818188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:50.818190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:50.818282 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.818285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.818916 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.818917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.819709 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:50.819711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.820244 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.820245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:50.820316 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.820320 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:50.820367 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:01:50.820410 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":10,"thread_start_us":95,"threads_started":1}
I20260810 15:01:50.820890 29236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3313145c48fe477eb64cd6cff4d1298b" candidate_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c2960510514dc2987d21f40bccdbf7" is_pre_election: true
I20260810 15:01:50.820891 29488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3313145c48fe477eb64cd6cff4d1298b" candidate_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8029f68ab0b488dbb787be3be96ae3e" is_pre_election: true
I20260810 15:01:50.821043 29236 raft_consensus.cc:2454] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a88e0e49b8a4091a3105d6fe7ace519 in term 0.
I20260810 15:01:50.821043 29488 raft_consensus.cc:2454] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a88e0e49b8a4091a3105d6fe7ace519 in term 0.
I20260810 15:01:50.821270 28891 leader_election.cc:302] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a88e0e49b8a4091a3105d6fe7ace519, 74c2960510514dc2987d21f40bccdbf7; no voters:
I20260810 15:01:50.821401 29602 raft_consensus.cc:2781] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:50.821460 29602 raft_consensus.cc:482] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:50.821475 29602 raft_consensus.cc:3037] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:50.822202 29602 raft_consensus.cc:504] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.822326 29602 leader_election.cc:288] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [CANDIDATE]: Term 1 election: Requested vote from peers c8029f68ab0b488dbb787be3be96ae3e (127.28.193.195:45121), 74c2960510514dc2987d21f40bccdbf7 (127.28.193.194:38331)
I20260810 15:01:50.822464 29488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3313145c48fe477eb64cd6cff4d1298b" candidate_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8029f68ab0b488dbb787be3be96ae3e"
I20260810 15:01:50.822535 29236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3313145c48fe477eb64cd6cff4d1298b" candidate_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c2960510514dc2987d21f40bccdbf7"
I20260810 15:01:50.822547 29488 raft_consensus.cc:3037] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:50.822603 29236 raft_consensus.cc:3037] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:50.823411 29236 raft_consensus.cc:2454] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a88e0e49b8a4091a3105d6fe7ace519 in term 1.
I20260810 15:01:50.823411 29488 raft_consensus.cc:2454] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a88e0e49b8a4091a3105d6fe7ace519 in term 1.
I20260810 15:01:50.823581 28892 leader_election.cc:302] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a88e0e49b8a4091a3105d6fe7ace519, c8029f68ab0b488dbb787be3be96ae3e; no voters:
I20260810 15:01:50.823662 29602 raft_consensus.cc:2781] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:50.823853 29602 raft_consensus.cc:686] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [term 1 LEADER]: Becoming Leader. State: Replica: 2a88e0e49b8a4091a3105d6fe7ace519, State: Running, Role: LEADER
I20260810 15:01:50.823945 29602 consensus_queue.cc:237] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } }
I20260810 15:01:50.824756 28762 catalog_manager.cc:5495] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a88e0e49b8a4091a3105d6fe7ace519 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a88e0e49b8a4091a3105d6fe7ace519" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 42065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:50.829193 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.827113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.28.193.193:42065)
0810 15:01:50.827155 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:50.827156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:50.827164 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:01:50.827225 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:50.827226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:50.827239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:50.827326 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:50.827430 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.827847 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.828532 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:50.828719 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:50.828782 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:50.828963 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:50.829054 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:50.829111 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:50.829126 (+ 15us) server_negotiation.cc:299] Negotiation successful
0810 15:01:50.829142 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:01:50.830778 29236 raft_consensus.cc:1261] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 1 FOLLOWER]: Refusing update from remote peer 2a88e0e49b8a4091a3105d6fe7ace519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:50.830794 29488 raft_consensus.cc:1261] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 1 FOLLOWER]: Refusing update from remote peer 2a88e0e49b8a4091a3105d6fe7ace519: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:50.831050 29602 consensus_queue.cc:1035] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c2960510514dc2987d21f40bccdbf7" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 38331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:50.831182 29613 consensus_queue.cc:1035] T 3313145c48fe477eb64cd6cff4d1298b P 2a88e0e49b8a4091a3105d6fe7ace519 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8029f68ab0b488dbb787be3be96ae3e" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:50.835500 28761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:50.835597 28761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:50.836284 28761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e160815a405641399035e5b09a769311: Sending DeleteTablet for 3 replicas of tablet 3313145c48fe477eb64cd6cff4d1298b
I20260810 15:01:50.836620 29207 tablet_service.cc:1514] Processing DeleteTablet for tablet 3313145c48fe477eb64cd6cff4d1298b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:50 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:57054
I20260810 15:01:50.836664 29441 tablet_service.cc:1514] Processing DeleteTablet for tablet 3313145c48fe477eb64cd6cff4d1298b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:50 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:45966
I20260810 15:01:50.836696 28974 tablet_service.cc:1514] Processing DeleteTablet for tablet 3313145c48fe477eb64cd6cff4d1298b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:50 UTC) from {username='kudu', principal='kudu/127.28.193.254@KRBTEST.COM'} at 127.0.0.1:39152
I20260810 15:01:50.836874 29631 tablet_replica.cc:330] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: stopping tablet replica
I20260810 15:01:50.836971 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 28868
I20260810 15:01:50.866727 29631 raft_consensus.cc:2229] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:50.866918 29631 raft_consensus.cc:2258] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:50.867928 29633 tablet_replica.cc:330] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: stopping tablet replica
I20260810 15:01:50.868024 29631 ts_tablet_manager.cc:1905] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:50.868063 29633 raft_consensus.cc:2229] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:50.868175 29633 raft_consensus.cc:2258] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:50.869371 29633 ts_tablet_manager.cc:1905] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:50.869558 29631 ts_tablet_manager.cc:1918] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:50.869642 29631 log.cc:1196] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/3313145c48fe477eb64cd6cff4d1298b
I20260810 15:01:50.869930 29631 ts_tablet_manager.cc:1939] T 3313145c48fe477eb64cd6cff4d1298b P 74c2960510514dc2987d21f40bccdbf7: Deleting consensus metadata
I20260810 15:01:50.870420 28735 catalog_manager.cc:4841] TS 74c2960510514dc2987d21f40bccdbf7 (127.28.193.194:38331): tablet 3313145c48fe477eb64cd6cff4d1298b (table test-table [id=40e55904374d4940865922dcf6822ba0]) successfully deleted
I20260810 15:01:50.870831 29633 ts_tablet_manager.cc:1918] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:50.870898 29633 log.cc:1196] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/3313145c48fe477eb64cd6cff4d1298b
I20260810 15:01:50.871145 29633 ts_tablet_manager.cc:1939] T 3313145c48fe477eb64cd6cff4d1298b P c8029f68ab0b488dbb787be3be96ae3e: Deleting consensus metadata
I20260810 15:01:50.871542 28736 catalog_manager.cc:4841] TS c8029f68ab0b488dbb787be3be96ae3e (127.28.193.195:45121): tablet 3313145c48fe477eb64cd6cff4d1298b (table test-table [id=40e55904374d4940865922dcf6822ba0]) successfully deleted
W20260810 15:01:50.873430 28736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:42065 (error 108)
I20260810 15:01:50.873956 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:50.873826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:42065 (local address 127.0.0.1:39156)
0810 15:01:50.873878 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:50.873925 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:42065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:01:50.873993 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29096
W20260810 15:01:50.874075 28736 catalog_manager.cc:4584] TS 2a88e0e49b8a4091a3105d6fe7ace519 (127.28.193.193:42065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3313145c48fe477eb64cd6cff4d1298b: Network error: Client connection negotiation failed: client connection to 127.28.193.193:42065: connect: Connection refused (error 111)
I20260810 15:01:50.881188 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29327
I20260810 15:01:50.893486 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 28708
2026-08-10T15:01:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2218 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T15:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:50Z Disabled control of system clock
I20260810 15:01:50.924742 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34325
--webserver_interface=127.28.193.254
--webserver_port=0
--builtin_ntp_servers=127.28.193.212:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:51.004746 29639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:51.004954 29639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:51.004973 29639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:51.006232 29639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:51.006273 29639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:51.006284 29639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:51.006295 29639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:51.006306 29639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:51.007642 29639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:51.007871 29639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:51.008137 29639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:51.012735 29653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.012707 29656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.012719 29650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.012948 29639 server_base.cc:1034] running on GCE node
I20260810 15:01:51.013269 29639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:51.013548 29639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:51.014765 29639 hybrid_clock.cc:648] HybridClock initialized: now 1786374111014720 us; error 54 us; skew 500 ppm
I20260810 15:01:51.015981 29639 webserver.cc:460] Webserver started at http://127.28.193.254:45293/ using document root <none> and password file <none>
I20260810 15:01:51.016203 29639 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:51.016253 29639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:51.016389 29639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:51.017473 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "2adcc565e37a4f049b60948091ed34e4"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "7977aa70fb181181a58b3eac9a4698cc"
server_key_iv: "d2236fb963af8fb18434680742cbd4a3"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.017830 29639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "2adcc565e37a4f049b60948091ed34e4"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "7977aa70fb181181a58b3eac9a4698cc"
server_key_iv: "d2236fb963af8fb18434680742cbd4a3"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.019084 29639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:01:51.019778 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.019958 29639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:51.020038 29639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "2adcc565e37a4f049b60948091ed34e4"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "7977aa70fb181181a58b3eac9a4698cc"
server_key_iv: "d2236fb963af8fb18434680742cbd4a3"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.020099 29639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:51.048868 29639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:51.049202 29639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:51.049341 29639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:51.053452 29639 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34325
I20260810 15:01:51.053500 29747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34325 every 8 connection(s)
I20260810 15:01:51.053846 29639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:51.054840 29749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:51.057138 29749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: Bootstrap starting.
I20260810 15:01:51.057777 29749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:51.058053 29749 log.cc:824] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:51.058720 29749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: No bootstrap required, opened a new log
I20260810 15:01:51.059154 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29639
I20260810 15:01:51.059262 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:51.059427 29447 external_mini_cluster.cc:1428] Setting key 535d805ad1323bab8fa11486b06cb2e6
I20260810 15:01:51.060658 29749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } }
I20260810 15:01:51.060832 29749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:51.060868 29749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adcc565e37a4f049b60948091ed34e4, State: Initialized, Role: FOLLOWER
I20260810 15:01:51.060982 29749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } }
I20260810 15:01:51.061061 29749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:51.061090 29749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:51.061137 29749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:51.061851 29749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } }
I20260810 15:01:51.061962 29749 leader_election.cc:302] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2adcc565e37a4f049b60948091ed34e4; no voters:
I20260810 15:01:51.062131 29749 leader_election.cc:288] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:51.062180 29760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:51.062322 29760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2adcc565e37a4f049b60948091ed34e4, State: Running, Role: LEADER
I20260810 15:01:51.062376 29749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:51.062417 29760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } }
I20260810 15:01:51.062747 29760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2adcc565e37a4f049b60948091ed34e4. Latest consensus state: current_term: 1 leader_uuid: "2adcc565e37a4f049b60948091ed34e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } } }
I20260810 15:01:51.062848 29760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:51.062749 29764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2adcc565e37a4f049b60948091ed34e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2adcc565e37a4f049b60948091ed34e4" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34325 } } }
I20260810 15:01:51.063005 29764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:51.063134 29769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:51.063656 29769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:51.065462 29769 catalog_manager.cc:1347] Generated new cluster ID: c11f1d21d16649d8b8f29883ccc828db
I20260810 15:01:51.065508 29769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:51.066051 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.059943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.28.193.254:34325)
0810 15:01:51.060170 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.060180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.060216 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:01:51.060628 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.060637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.060715 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.060869 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:51.061111 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.062876 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.063972 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.064411 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.064521 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.065215 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:51.065219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:51.065220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:51.065428 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:51.065467 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:51.065483 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:51.065549 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.065774 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.065851 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260810 15:01:51.075374 29769 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:51.075832 29769 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:51.082432 29769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: Generated new TSK 0
I20260810 15:01:51.082620 29769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:51.085811 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.193:0
--local_ip_for_outbound_sockets=127.28.193.193
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--builtin_ntp_servers=127.28.193.212:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:51.167135 29790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:51.167362 29790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:51.167397 29790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:51.168509 29790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:51.168546 29790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:51.168577 29790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.193
I20260810 15:01:51.169950 29790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.193.193
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.193
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:51.170178 29790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:51.170465 29790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:51.176312 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.176420 29790 server_base.cc:1034] running on GCE node
W20260810 15:01:51.176340 29800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.176501 29795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.176721 29790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:51.176966 29790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:51.178103 29790 hybrid_clock.cc:648] HybridClock initialized: now 1786374111178089 us; error 33 us; skew 500 ppm
I20260810 15:01:51.179288 29790 webserver.cc:460] Webserver started at http://127.28.193.193:37769/ using document root <none> and password file <none>
I20260810 15:01:51.179509 29790 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:51.179557 29790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:51.179687 29790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:51.180758 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data/instance:
uuid: "e928806c36244e5888ad2af933cdd8b0"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "5431f4d290d51d3e214e8eedd9fd062f"
server_key_iv: "81e48ad60c488a699a92bc8025fa9e03"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.181103 29790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance:
uuid: "e928806c36244e5888ad2af933cdd8b0"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "5431f4d290d51d3e214e8eedd9fd062f"
server_key_iv: "81e48ad60c488a699a92bc8025fa9e03"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.182803 29790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:51.183533 29812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.183688 29790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:51.183779 29790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal
uuid: "e928806c36244e5888ad2af933cdd8b0"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "5431f4d290d51d3e214e8eedd9fd062f"
server_key_iv: "81e48ad60c488a699a92bc8025fa9e03"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.183858 29790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:51.192137 29790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:51.192392 29790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:51.192512 29790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:51.192705 29790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:51.192999 29790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:51.193032 29790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.193058 29790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:51.193084 29790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.198309 29790 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.193:37477
I20260810 15:01:51.198369 30006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.193:37477 every 8 connection(s)
I20260810 15:01:51.198652 29790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/data/info.pb
I20260810 15:01:51.201136 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 29790
I20260810 15:01:51.201236 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-0/wal/instance
I20260810 15:01:51.201416 29447 external_mini_cluster.cc:1428] Setting key 7e1bdef8baff37140b64a4c7f3d72c05
I20260810 15:01:51.202363 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.194:0
--local_ip_for_outbound_sockets=127.28.193.194
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--builtin_ntp_servers=127.28.193.212:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:01:51.203696 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.199298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:33253 (local address 127.28.193.254:34325)
0810 15:01:51.199341 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.199343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.199526 (+ 183us) server_negotiation.cc:407] Connection header received
0810 15:01:51.200042 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.200044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.200057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.200108 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:51.200696 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.201184 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.202356 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.202617 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.202675 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.203438 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:51.203439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:51.203440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:51.203447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:51.203462 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:51.203466 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:51.203512 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.203590 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.203618 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:01:51.203913 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.199263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34325 (local address 127.28.193.193:33253)
0810 15:01:51.199426 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.199456 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.199947 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.200161 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.200164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.200172 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:51.200636 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.200640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.201278 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.201279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.202278 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.202281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.202693 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.202694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.202834 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.203305 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:51.203305 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:51.203332 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:51.203355 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:51.203518 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:51.203523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:51.203526 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.203584 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.203683 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260810 15:01:51.204355 30011 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34325
I20260810 15:01:51.204449 30011 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:51.204627 30011 heartbeater.cc:511] Master 127.28.193.254:34325 requested a full tablet report, sending...
I20260810 15:01:51.205155 29690 ts_manager.cc:194] Registered new tserver with Master: e928806c36244e5888ad2af933cdd8b0 (127.28.193.193:37477)
I20260810 15:01:51.205729 29690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.193:33253
W20260810 15:01:51.284310 30017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:51.284509 30017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:51.284529 30017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:51.285604 30017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:51.285648 30017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:51.285676 30017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.194
I20260810 15:01:51.287241 30017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.193.194
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.194
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:51.287444 30017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:51.287683 30017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:51.292626 30023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.292615 30025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.292737 30017 server_base.cc:1034] running on GCE node
W20260810 15:01:51.292598 30027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.292963 30017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:51.293176 30017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:51.294308 30017 hybrid_clock.cc:648] HybridClock initialized: now 1786374111294298 us; error 26 us; skew 500 ppm
I20260810 15:01:51.295432 30017 webserver.cc:460] Webserver started at http://127.28.193.194:38609/ using document root <none> and password file <none>
I20260810 15:01:51.295619 30017 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:51.295660 30017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:51.295751 30017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:51.296720 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data/instance:
uuid: "e172d738ceb94b8e86f8a2e013347e11"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4fb286b19142679e325167c9ac4aed47"
server_key_iv: "760923d69892e849f83fcda21d3267fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.297010 30017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance:
uuid: "e172d738ceb94b8e86f8a2e013347e11"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4fb286b19142679e325167c9ac4aed47"
server_key_iv: "760923d69892e849f83fcda21d3267fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.298235 30017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:01:51.298995 30037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.299149 30017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:51.299233 30017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal
uuid: "e172d738ceb94b8e86f8a2e013347e11"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4fb286b19142679e325167c9ac4aed47"
server_key_iv: "760923d69892e849f83fcda21d3267fa"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.299295 30017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:51.322552 30017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:51.322872 30017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:51.322989 30017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:51.323215 30017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:51.323565 30017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:51.323601 30017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.323622 30017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:51.323633 30017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.329073 30017 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.194:36259
I20260810 15:01:51.329134 30226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.194:36259 every 8 connection(s)
I20260810 15:01:51.329442 30017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/data/info.pb
I20260810 15:01:51.334576 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.330218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.194:60237 (local address 127.28.193.254:34325)
0810 15:01:51.330293 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.330295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.330511 (+ 216us) server_negotiation.cc:407] Connection header received
0810 15:01:51.331037 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.331038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.331051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.331104 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:51.331786 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.332302 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.333264 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.333467 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.333523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.334317 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:51.334319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:51.334319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:51.334324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:51.334340 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:51.334344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:51.334389 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.334465 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.334493 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:01:51.334810 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.330181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34325 (local address 127.28.193.194:60237)
0810 15:01:51.330386 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.330426 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.330937 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.331179 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.331183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.331194 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:51.331698 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.331702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.332389 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.332390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.333178 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.333180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.333540 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.333541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.333670 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.334190 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:51.334191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:51.334219 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:51.334243 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:51.334394 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:51.334400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:51.334403 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.334465 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.334565 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260810 15:01:51.335279 30228 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34325
I20260810 15:01:51.335366 30228 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:51.335541 30228 heartbeater.cc:511] Master 127.28.193.254:34325 requested a full tablet report, sending...
I20260810 15:01:51.335939 29690 ts_manager.cc:194] Registered new tserver with Master: e172d738ceb94b8e86f8a2e013347e11 (127.28.193.194:36259)
I20260810 15:01:51.336301 29690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.194:60237
I20260810 15:01:51.339197 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30017
I20260810 15:01:51.339293 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal/instance
I20260810 15:01:51.339464 29447 external_mini_cluster.cc:1428] Setting key 6598ac9bbb684db4187b4de38660c76d
I20260810 15:01:51.340445 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.193.195:0
--local_ip_for_outbound_sockets=127.28.193.195
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--builtin_ntp_servers=127.28.193.212:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:01:51.422441 30235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:51.422659 30235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:51.422690 30235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:51.423866 30235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:51.423904 30235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:51.423936 30235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.193.195
I20260810 15:01:51.425239 30235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.193.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.193.195
--webserver_port=0
--tserver_master_addrs=127.28.193.254:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.193.195
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:51.425482 30235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:51.425789 30235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:51.430625 30247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.430711 30244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:51.430652 30240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:51.430760 30235 server_base.cc:1034] running on GCE node
I20260810 15:01:51.430998 30235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:51.431210 30235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:51.432363 30235 hybrid_clock.cc:648] HybridClock initialized: now 1786374111432352 us; error 34 us; skew 500 ppm
I20260810 15:01:51.433647 30235 webserver.cc:460] Webserver started at http://127.28.193.195:36411/ using document root <none> and password file <none>
I20260810 15:01:51.433933 30235 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:51.433986 30235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:51.434109 30235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:51.435189 30235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data/instance:
uuid: "4e21cc53962e4d21ae959f65ecbad1ef"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4e73968c642f4813b2b25e98ff6c27ba"
server_key_iv: "75a305585279d8e454bbba0fd0234fd2"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.435527 30235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance:
uuid: "4e21cc53962e4d21ae959f65ecbad1ef"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4e73968c642f4813b2b25e98ff6c27ba"
server_key_iv: "75a305585279d8e454bbba0fd0234fd2"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.436836 30235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:01:51.437546 30263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.437701 30235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:51.437821 30235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal
uuid: "4e21cc53962e4d21ae959f65ecbad1ef"
format_stamp: "Formatted at 2026-08-10 15:01:51 on dist-test-slave-cvwc"
server_key: "4e73968c642f4813b2b25e98ff6c27ba"
server_key_iv: "75a305585279d8e454bbba0fd0234fd2"
server_key_version: "encryptionkey@0"
I20260810 15:01:51.437884 30235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:51.447052 30235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:51.447337 30235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:51.447466 30235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:51.447671 30235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:01:51.447997 30235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:01:51.448031 30235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.448058 30235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:01:51.448076 30235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:51.453699 30235 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.195:45757
I20260810 15:01:51.453778 30456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.195:45757 every 8 connection(s)
I20260810 15:01:51.454097 30235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/data/info.pb
I20260810 15:01:51.455786 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30235
I20260810 15:01:51.455904 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal/instance
I20260810 15:01:51.456127 29447 external_mini_cluster.cc:1428] Setting key 6459bca64e056239989874b2d5460d90
I20260810 15:01:51.459487 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.454815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.195:39761 (local address 127.28.193.254:34325)
0810 15:01:51.454891 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.454893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.455070 (+ 177us) server_negotiation.cc:407] Connection header received
0810 15:01:51.455604 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.455606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.455617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.455672 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:51.456580 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.457124 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.458149 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.458357 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.458414 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.459221 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:51.459222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:51.459223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:51.459227 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:51.459242 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:51.459246 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:51.459296 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.459398 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.459427 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:01:51.459621 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.454764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34325 (local address 127.28.193.195:39761)
0810 15:01:51.454953 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.454992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.455483 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.455753 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.455759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.455778 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:01:51.456430 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.456436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.457204 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.457205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.458072 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.458075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.458427 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.458428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.458560 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.459093 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:01:51.459094 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:01:51.459122 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:01:51.459146 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:01:51.459303 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:01:51.459306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:01:51.459309 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.459356 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.459436 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260810 15:01:51.460126 30458 heartbeater.cc:347] Connected to a master server at 127.28.193.254:34325
I20260810 15:01:51.460245 30458 heartbeater.cc:464] Registering TS with master...
I20260810 15:01:51.460433 30458 heartbeater.cc:511] Master 127.28.193.254:34325 requested a full tablet report, sending...
I20260810 15:01:51.460816 29690 ts_manager.cc:194] Registered new tserver with Master: 4e21cc53962e4d21ae959f65ecbad1ef (127.28.193.195:45757)
I20260810 15:01:51.461158 29690 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.193.195:39761
I20260810 15:01:51.467974 29447 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:01:51.472174 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.469373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.28.193.254:34325)
0810 15:01:51.469438 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.469440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.469506 (+ 66us) server_negotiation.cc:407] Connection header received
0810 15:01:51.469569 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.469570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.469582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.469624 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:51.469832 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.470312 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.471043 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.471228 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.471281 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.471942 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:51.471949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:51.471950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:51.471954 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:51.471972 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:51.471976 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:51.472017 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.472068 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.472094 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:01:51.473789 29690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:01:51.475358 29690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:01:51.481431 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.28.193.195:45757)
0810 15:01:51.477936 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.477941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.477952 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:01:51.478019 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.478021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.478038 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.478121 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:51.478310 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479068 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.480338 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481021 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481092 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.481297 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.481336 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":139,"threads_started":1}
I20260810 15:01:51.481424 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:45757 (local address 127.0.0.1:46994)
0810 15:01:51.477815 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.477857 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.477930 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.478177 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.478180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.478182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:51.478249 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.478251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479162 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.479163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.480225 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.480228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481104 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.481187 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481201 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.481276 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.481335 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"spinlock_wait_cycles":768,"thread_start_us":110,"threads_started":1}
I20260810 15:01:51.481806 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36259 (local address 127.0.0.1:38292)
0810 15:01:51.478045 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.478054 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.478108 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.478513 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.478514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.478516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:51.478549 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.478550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479752 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.479753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.480607 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.480610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481543 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.481607 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481611 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.481660 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.481700 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":98,"threads_started":1}
I20260810 15:01:51.481841 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.28.193.194:36259)
0810 15:01:51.477982 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.477987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.478325 (+ 338us) server_negotiation.cc:407] Connection header received
0810 15:01:51.478367 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.478369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.478384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.478602 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:51.478662 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479684 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.480677 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481353 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481426 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481502 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.481682 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.481756 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260810 15:01:51.482435 30347 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 (DEFAULT_TABLE table=test-table [id=5b30b74a19f04bdca7b517aa32ddc587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:51.482457 30119 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 (DEFAULT_TABLE table=test-table [id=5b30b74a19f04bdca7b517aa32ddc587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:51.482846 30119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b1adf8a8154b2196b8b90ceb1c56e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:51.482846 30347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b1adf8a8154b2196b8b90ceb1c56e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:51.485595 30486 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Bootstrap starting.
I20260810 15:01:51.485761 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.28.193.193:37477)
0810 15:01:51.478058 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.478061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.478090 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:01:51.478131 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.478133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.478148 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.478222 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:51.478444 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479572 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.480861 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481536 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481597 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481672 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.485581 (+ 3909us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.485637 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":96,"threads_started":1}
I20260810 15:01:51.485679 30484 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Bootstrap starting.
I20260810 15:01:51.485944 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.477496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:37477 (local address 127.0.0.1:54428)
0810 15:01:51.477886 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.477898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.477950 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.478291 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.478292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.478293 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:51.478334 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.478335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.479668 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.479669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.480773 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.480777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.481703 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.481704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.481788 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.481791 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.481836 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.481870 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":43,"thread_start_us":61,"threads_started":1}
I20260810 15:01:51.486356 30486 tablet_bootstrap.cc:654] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:51.486470 30484 tablet_bootstrap.cc:654] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:51.486759 30484 log.cc:824] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:51.486759 30486 log.cc:824] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:51.486796 29894 tablet_service.cc:1467] Processing CreateTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 (DEFAULT_TABLE table=test-table [id=5b30b74a19f04bdca7b517aa32ddc587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:01:51.487166 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b1adf8a8154b2196b8b90ceb1c56e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:51.487535 30486 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: No bootstrap required, opened a new log
I20260810 15:01:51.487572 30484 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: No bootstrap required, opened a new log
I20260810 15:01:51.487596 30486 ts_tablet_manager.cc:1397] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:01:51.487635 30484 ts_tablet_manager.cc:1397] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:51.489480 30486 raft_consensus.cc:348] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.489612 30486 raft_consensus.cc:374] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:51.489646 30486 raft_consensus.cc:729] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21cc53962e4d21ae959f65ecbad1ef, State: Initialized, Role: FOLLOWER
I20260810 15:01:51.489818 30486 consensus_queue.cc:260] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.490085 30486 ts_tablet_manager.cc:1428] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 15:01:51.490113 30458 heartbeater.cc:503] Master 127.28.193.254:34325 was elected leader, sending a full tablet report...
I20260810 15:01:51.490171 30484 raft_consensus.cc:348] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.490362 30484 raft_consensus.cc:374] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:51.490417 30484 raft_consensus.cc:729] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e172d738ceb94b8e86f8a2e013347e11, State: Initialized, Role: FOLLOWER
I20260810 15:01:51.490546 30484 consensus_queue.cc:260] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.490808 30484 ts_tablet_manager.cc:1428] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:01:51.490942 30228 heartbeater.cc:503] Master 127.28.193.254:34325 was elected leader, sending a full tablet report...
I20260810 15:01:51.491111 30493 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: Bootstrap starting.
I20260810 15:01:51.491677 30493 tablet_bootstrap.cc:654] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:51.491961 30493 log.cc:824] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:51.492660 30493 tablet_bootstrap.cc:492] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: No bootstrap required, opened a new log
I20260810 15:01:51.492717 30493 ts_tablet_manager.cc:1397] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:01:51.494470 30493 raft_consensus.cc:348] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.494588 30493 raft_consensus.cc:374] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:51.494608 30493 raft_consensus.cc:729] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e928806c36244e5888ad2af933cdd8b0, State: Initialized, Role: FOLLOWER
I20260810 15:01:51.494704 30493 consensus_queue.cc:260] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.494961 30493 ts_tablet_manager.cc:1428] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:01:51.495026 30011 heartbeater.cc:503] Master 127.28.193.254:34325 was elected leader, sending a full tablet report...
I20260810 15:01:51.530037 30500 raft_consensus.cc:482] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:01:51.530184 30500 raft_consensus.cc:504] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.530556 30500 leader_election.cc:288] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e21cc53962e4d21ae959f65ecbad1ef (127.28.193.195:45757), e172d738ceb94b8e86f8a2e013347e11 (127.28.193.194:36259)
I20260810 15:01:51.534493 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.530908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:60871 (local address 127.28.193.194:36259)
0810 15:01:51.530962 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.530964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.530975 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:01:51.531010 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.531010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.531019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.531089 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:51.531236 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.532012 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.533515 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.534115 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.534187 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.534196 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.534371 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.534418 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:01:51.534493 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.530881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.193.193:56577 (local address 127.28.193.195:45757)
0810 15:01:51.530935 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.530937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.531085 (+ 148us) server_negotiation.cc:407] Connection header received
0810 15:01:51.531123 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.531124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.531134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.531195 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:01:51.531326 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.532011 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.533515 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.534109 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.534187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.534196 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.534377 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.534420 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:01:51.534502 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.530785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.194:36259 (local address 127.28.193.193:60871)
0810 15:01:51.530859 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.530872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.530928 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.531094 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.531095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.531096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:51.531164 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.531166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.532110 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.532112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.533403 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.533406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.534198 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.534200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.534296 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.534298 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.534357 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.534405 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":45}
I20260810 15:01:51.534507 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.530788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.195:45757 (local address 127.28.193.193:56577)
0810 15:01:51.531004 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.531013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:01:51.531060 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:01:51.531203 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:01:51.531203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:01:51.531205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:01:51.531263 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.531264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.532110 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.532112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.533403 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:01:51.533406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.534211 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.534211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:01:51.534292 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.534297 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:01:51.534357 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:01:51.534404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":64,"thread_start_us":137,"threads_started":1}
I20260810 15:01:51.534940 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b1adf8a8154b2196b8b90ceb1c56e8" candidate_uuid: "e928806c36244e5888ad2af933cdd8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" is_pre_election: true
I20260810 15:01:51.534929 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b1adf8a8154b2196b8b90ceb1c56e8" candidate_uuid: "e928806c36244e5888ad2af933cdd8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e172d738ceb94b8e86f8a2e013347e11" is_pre_election: true
I20260810 15:01:51.535077 30157 raft_consensus.cc:2454] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e928806c36244e5888ad2af933cdd8b0 in term 0.
I20260810 15:01:51.535076 30382 raft_consensus.cc:2454] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e928806c36244e5888ad2af933cdd8b0 in term 0.
I20260810 15:01:51.535300 29816 leader_election.cc:302] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e21cc53962e4d21ae959f65ecbad1ef, e928806c36244e5888ad2af933cdd8b0; no voters:
I20260810 15:01:51.535465 30500 raft_consensus.cc:2781] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:01:51.535511 30500 raft_consensus.cc:482] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:01:51.535534 30500 raft_consensus.cc:3037] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:51.536264 30500 raft_consensus.cc:504] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.536383 30500 leader_election.cc:288] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [CANDIDATE]: Term 1 election: Requested vote from peers 4e21cc53962e4d21ae959f65ecbad1ef (127.28.193.195:45757), e172d738ceb94b8e86f8a2e013347e11 (127.28.193.194:36259)
I20260810 15:01:51.536520 30382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b1adf8a8154b2196b8b90ceb1c56e8" candidate_uuid: "e928806c36244e5888ad2af933cdd8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e21cc53962e4d21ae959f65ecbad1ef"
I20260810 15:01:51.536530 30157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b1adf8a8154b2196b8b90ceb1c56e8" candidate_uuid: "e928806c36244e5888ad2af933cdd8b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e172d738ceb94b8e86f8a2e013347e11"
I20260810 15:01:51.536602 30382 raft_consensus.cc:3037] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:51.536615 30157 raft_consensus.cc:3037] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:51.537272 30157 raft_consensus.cc:2454] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e928806c36244e5888ad2af933cdd8b0 in term 1.
I20260810 15:01:51.537325 30382 raft_consensus.cc:2454] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e928806c36244e5888ad2af933cdd8b0 in term 1.
I20260810 15:01:51.537447 29813 leader_election.cc:302] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e172d738ceb94b8e86f8a2e013347e11, e928806c36244e5888ad2af933cdd8b0; no voters:
I20260810 15:01:51.537539 30500 raft_consensus.cc:2781] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:51.537667 30500 raft_consensus.cc:686] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 1 LEADER]: Becoming Leader. State: Replica: e928806c36244e5888ad2af933cdd8b0, State: Running, Role: LEADER
I20260810 15:01:51.537770 30500 consensus_queue.cc:237] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } }
I20260810 15:01:51.538547 29690 catalog_manager.cc:5495] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e928806c36244e5888ad2af933cdd8b0 (127.28.193.193). New cstate: current_term: 1 leader_uuid: "e928806c36244e5888ad2af933cdd8b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e928806c36244e5888ad2af933cdd8b0" member_type: VOTER last_known_addr { host: "127.28.193.193" port: 37477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 } health_report { overall_health: UNKNOWN } } }
I20260810 15:01:51.557209 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.555112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.28.193.193:37477)
0810 15:01:51.555165 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:51.555167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:51.555175 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:01:51.555209 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:51.555210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:51.555230 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:51.555302 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:01:51.555413 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.555885 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.556607 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:51.556796 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:51.556840 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:51.556948 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:01:51.557048 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:01:51.557104 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:51.557116 (+ 12us) server_negotiation.cc:299] Negotiation successful
0810 15:01:51.557132 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:01:51.559101 30157 raft_consensus.cc:1261] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 1 FOLLOWER]: Refusing update from remote peer e928806c36244e5888ad2af933cdd8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:51.559115 30382 raft_consensus.cc:1261] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 1 FOLLOWER]: Refusing update from remote peer e928806c36244e5888ad2af933cdd8b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:01:51.559435 30500 consensus_queue.cc:1035] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e172d738ceb94b8e86f8a2e013347e11" member_type: VOTER last_known_addr { host: "127.28.193.194" port: 36259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:51.559581 30502 consensus_queue.cc:1035] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21cc53962e4d21ae959f65ecbad1ef" member_type: VOTER last_known_addr { host: "127.28.193.195" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:01:51.562071 30506 mvcc.cc:205] Tried to move back new op lower bound from 7316988360943267840 to 7316988360859185152. Current Snapshot: MvccSnapshot[applied={T|T < 7316988360943267840}]
I20260810 15:01:51.565109 29688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:51.565197 29688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:01:51.565924 29688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2adcc565e37a4f049b60948091ed34e4: Sending DeleteTablet for 3 replicas of tablet 29b1adf8a8154b2196b8b90ceb1c56e8
I20260810 15:01:51.566231 29894 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:51 UTC) from {username='slave'} at 127.0.0.1:54428
I20260810 15:01:51.566378 30119 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:51 UTC) from {username='slave'} at 127.0.0.1:38292
I20260810 15:01:51.566452 30347 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b1adf8a8154b2196b8b90ceb1c56e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:01:51 UTC) from {username='slave'} at 127.0.0.1:46994
I20260810 15:01:51.566599 30521 tablet_replica.cc:330] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: stopping tablet replica
I20260810 15:01:51.566655 30520 tablet_replica.cc:330] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0: stopping tablet replica
I20260810 15:01:51.566691 30521 raft_consensus.cc:2229] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:51.566733 30520 raft_consensus.cc:2229] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:01:51.566788 30521 raft_consensus.cc:2258] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:51.566847 30520 raft_consensus.cc:2258] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e928806c36244e5888ad2af933cdd8b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:51.566859 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29790
I20260810 15:01:51.566975 30522 tablet_replica.cc:330] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: stopping tablet replica
I20260810 15:01:51.567245 30521 ts_tablet_manager.cc:1905] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:51.569144 30521 ts_tablet_manager.cc:1918] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:51.569204 30521 log.cc:1196] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-1/wal/wals/29b1adf8a8154b2196b8b90ceb1c56e8
I20260810 15:01:51.569394 30521 ts_tablet_manager.cc:1939] T 29b1adf8a8154b2196b8b90ceb1c56e8 P e172d738ceb94b8e86f8a2e013347e11: Deleting consensus metadata
I20260810 15:01:51.569759 29667 catalog_manager.cc:4841] TS e172d738ceb94b8e86f8a2e013347e11 (127.28.193.194:36259): tablet 29b1adf8a8154b2196b8b90ceb1c56e8 (table test-table [id=5b30b74a19f04bdca7b517aa32ddc587]) successfully deleted
I20260810 15:01:51.576838 30522 raft_consensus.cc:2229] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:01:51.577018 30522 raft_consensus.cc:2258] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:01:51.577488 30522 ts_tablet_manager.cc:1905] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:01:51.578896 30522 ts_tablet_manager.cc:1918] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:01:51.578969 30522 log.cc:1196] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Deleting WAL directory at /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786373869351853-29447-0/minicluster-data/ts-2/wal/wals/29b1adf8a8154b2196b8b90ceb1c56e8
I20260810 15:01:51.579192 30522 ts_tablet_manager.cc:1939] T 29b1adf8a8154b2196b8b90ceb1c56e8 P 4e21cc53962e4d21ae959f65ecbad1ef: Deleting consensus metadata
I20260810 15:01:51.579625 29670 catalog_manager.cc:4841] TS 4e21cc53962e4d21ae959f65ecbad1ef (127.28.193.195:45757): tablet 29b1adf8a8154b2196b8b90ceb1c56e8 (table test-table [id=5b30b74a19f04bdca7b517aa32ddc587]) successfully deleted
W20260810 15:01:51.581115 29668 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.193.193:37477 (error 108)
I20260810 15:01:51.581336 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30017
I20260810 15:01:51.594851 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:51.594660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.193:37477 (local address 127.0.0.1:54438)
0810 15:01:51.594732 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 15:01:51.594805 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.193:37477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 15:01:51.595018 29668 catalog_manager.cc:4584] TS e928806c36244e5888ad2af933cdd8b0 (127.28.193.193:37477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29b1adf8a8154b2196b8b90ceb1c56e8: Network error: Client connection negotiation failed: client connection to 127.28.193.193:37477: connect: Connection refused (error 111)
I20260810 15:01:51.597988 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30235
I20260810 15:01:51.608515 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 29639
2026-08-10T15:01:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (718 ms)
[----------] 6 tests from EncryptionPolicyTest (8857 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:51 dist-test-slave-cvwc krb5kdc[30528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:51 dist-test-slave-cvwc krb5kdc[30528](info): set up 2 sockets
Aug 10 15:01:51 dist-test-slave-cvwc krb5kdc[30528](info): commencing operation
krb5kdc: starting...
W20260810 15:01:52.809092 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:52 dist-test-slave-cvwc krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:01:52.928834 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:53.012804 30546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:53.013021 30546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:53.013054 30546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:53.014232 30546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:53.014279 30546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:53.014302 30546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:53.014322 30546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:53.014349 30546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:53.015854 30546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:53.016108 30546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:53.016323 30546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:53.021104 30556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:53.021099 30560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:53.021128 30557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:53.021391 30546 server_base.cc:1034] running on GCE node
I20260810 15:01:53.021665 30546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:53.021973 30546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:53.023110 30546 hybrid_clock.cc:648] HybridClock initialized: now 1786374113023099 us; error 28 us; skew 500 ppm
Aug 10 15:01:53 dist-test-slave-cvwc krb5kdc[30528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:53.025476 30546 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:01:53.025939 30546 webserver.cc:460] Webserver started at http://127.0.0.1:41507/ using document root <none> and password file <none>
I20260810 15:01:53.026167 30546 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:53.026216 30546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:53.026336 30546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:53.027379 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "edb85e5cd6384128a41a260f892ecfe0"
format_stamp: "Formatted at 2026-08-10 15:01:53 on dist-test-slave-cvwc"
server_key: "dcb7e974794fb43fd5328c3396c7cf9b"
server_key_iv: "e658f7a89880754adc657e39498a9c2a"
server_key_version: "encryptionkey@0"
I20260810 15:01:53.027716 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "edb85e5cd6384128a41a260f892ecfe0"
format_stamp: "Formatted at 2026-08-10 15:01:53 on dist-test-slave-cvwc"
server_key: "dcb7e974794fb43fd5328c3396c7cf9b"
server_key_iv: "e658f7a89880754adc657e39498a9c2a"
server_key_version: "encryptionkey@0"
I20260810 15:01:53.028990 30546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:01:53.029742 30571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:53.029910 30546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:01:53.029997 30546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "edb85e5cd6384128a41a260f892ecfe0"
format_stamp: "Formatted at 2026-08-10 15:01:53 on dist-test-slave-cvwc"
server_key: "dcb7e974794fb43fd5328c3396c7cf9b"
server_key_iv: "e658f7a89880754adc657e39498a9c2a"
server_key_version: "encryptionkey@0"
I20260810 15:01:53.030066 30546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:53.041546 30546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:53.043258 30546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:53.043392 30546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:53.047183 30546 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44973
I20260810 15:01:53.047219 30656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44973 every 8 connection(s)
I20260810 15:01:53.047537 30546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:53.048450 30660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:53.050855 30660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: Bootstrap starting.
I20260810 15:01:53.051460 30660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:53.051733 30660 log.cc:824] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:53.052515 30660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: No bootstrap required, opened a new log
I20260810 15:01:53.053992 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30546
I20260810 15:01:53.054108 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:53.054280 29447 external_mini_cluster.cc:1428] Setting key f69dc35e53659e15ff18a619bcede5b1
I20260810 15:01:53.054205 30660 raft_consensus.cc:348] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260810 15:01:53.054364 30660 raft_consensus.cc:374] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:53.054414 30660 raft_consensus.cc:729] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb85e5cd6384128a41a260f892ecfe0, State: Initialized, Role: FOLLOWER
I20260810 15:01:53.054565 30660 consensus_queue.cc:260] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260810 15:01:53.054630 30660 raft_consensus.cc:388] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:53.054665 30660 raft_consensus.cc:482] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:53.054711 30660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:53.055616 30660 raft_consensus.cc:504] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260810 15:01:53.055712 30660 leader_election.cc:302] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edb85e5cd6384128a41a260f892ecfe0; no voters:
I20260810 15:01:53.055877 30660 leader_election.cc:288] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:53.055927 30667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:53.056053 30667 raft_consensus.cc:686] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [term 1 LEADER]: Becoming Leader. State: Replica: edb85e5cd6384128a41a260f892ecfe0, State: Running, Role: LEADER
I20260810 15:01:53.056136 30667 consensus_queue.cc:237] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } }
I20260810 15:01:53.056159 30660 sys_catalog.cc:564] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:53.057277 30670 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edb85e5cd6384128a41a260f892ecfe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } } }
I20260810 15:01:53.057482 30669 sys_catalog.cc:455] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edb85e5cd6384128a41a260f892ecfe0. Latest consensus state: current_term: 1 leader_uuid: "edb85e5cd6384128a41a260f892ecfe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edb85e5cd6384128a41a260f892ecfe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44973 } } }
I20260810 15:01:53.057741 30670 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:53.057889 30669 sys_catalog.cc:458] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0 [sys.catalog]: This master's current role is: LEADER
W20260810 15:01:53.058087 30688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:01:53.058135 30688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 15:01:53 dist-test-slave-cvwc krb5kdc[30528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:01:53.061858 30692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:53.062074 30692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:53.063870 30692 catalog_manager.cc:1347] Generated new cluster ID: 3e3eb1159e3b4c7f968f982fcbfd3bac
I20260810 15:01:53.063922 30692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:53.065006 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:53.054836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54757 (local address 127.0.0.1:44973)
0810 15:01:53.055083 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:53.055094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:53.055132 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:01:53.055256 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:53.055262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:53.055307 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:53.055423 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:53.056192 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:53.057384 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:53.058488 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:53.058903 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:53.058999 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:53.061331 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:53.061336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:53.061337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:53.061350 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:53.063210 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:53.063625 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:53.063629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:53.063629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:53.063753 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:53.064051 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:53.064052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:53.064052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:53.064376 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:53.064568 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:53.064681 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:01:53.064795 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260810 15:01:53.065939 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:01:53.074169 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:53.067324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47097 (local address 127.0.0.1:44973)
0810 15:01:53.067356 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:53.067358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:53.067493 (+ 135us) server_negotiation.cc:407] Connection header received
0810 15:01:53.067521 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:53.067522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:53.067533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:53.067592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:53.068156 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:53.068917 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:53.069855 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:53.070217 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:53.070287 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:53.071539 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:53.071541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:53.071541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:53.071548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:53.073030 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:53.073400 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:53.073401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:53.073401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:53.073456 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:53.073648 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:53.073649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:53.073649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:53.073803 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:53.073898 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:53.074013 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 15:01:53.074077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:01:53.092643 30692 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:53.093196 30692 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:53.102113 30692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edb85e5cd6384128a41a260f892ecfe0: Generated new TSK 0
I20260810 15:01:53.102319 30692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:53.111001 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30546
W20260810 15:01:53.124222 30525 connection.cc:535] client connection to 127.0.0.1:44973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44973): Connection reset by peer (error 104)
2026-08-10T15:01:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:53 dist-test-slave-cvwc krb5kdc[30707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:53 dist-test-slave-cvwc krb5kdc[30707](info): set up 2 sockets
Aug 10 15:01:53 dist-test-slave-cvwc krb5kdc[30707](info): commencing operation
krb5kdc: starting...
W20260810 15:01:54.282116 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:54 dist-test-slave-cvwc krb5kdc[30707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:01:54.397863 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:54.478121 30731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:54.478329 30731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:54.478351 30731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:54.479468 30731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:54.479504 30731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:54.479517 30731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:54.479528 30731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:54.479538 30731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:54.480954 30731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:54.481149 30731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:54.481331 30731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:54.486076 30741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:54.486099 30731 server_base.cc:1034] running on GCE node
W20260810 15:01:54.486099 30740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:54.486079 30743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:54.486516 30731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:54.486805 30731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:54.487951 30731 hybrid_clock.cc:648] HybridClock initialized: now 1786374114487935 us; error 28 us; skew 500 ppm
Aug 10 15:01:54 dist-test-slave-cvwc krb5kdc[30707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:54.490346 30731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:01:54.490790 30731 webserver.cc:460] Webserver started at http://127.0.0.1:36331/ using document root <none> and password file <none>
I20260810 15:01:54.490991 30731 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:54.491039 30731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:54.491228 30731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:54.492354 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "1888c922e3f0427b9300fb547a66ea58"
format_stamp: "Formatted at 2026-08-10 15:01:54 on dist-test-slave-cvwc"
server_key: "c8f2f486eee2d29a36ddf2e2368de758"
server_key_iv: "bf8d61e22eefdab4f605eb9465bbee13"
server_key_version: "encryptionkey@0"
I20260810 15:01:54.492693 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "1888c922e3f0427b9300fb547a66ea58"
format_stamp: "Formatted at 2026-08-10 15:01:54 on dist-test-slave-cvwc"
server_key: "c8f2f486eee2d29a36ddf2e2368de758"
server_key_iv: "bf8d61e22eefdab4f605eb9465bbee13"
server_key_version: "encryptionkey@0"
I20260810 15:01:54.494009 30731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:01:54.494719 30755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:54.494879 30731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:54.494963 30731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "1888c922e3f0427b9300fb547a66ea58"
format_stamp: "Formatted at 2026-08-10 15:01:54 on dist-test-slave-cvwc"
server_key: "c8f2f486eee2d29a36ddf2e2368de758"
server_key_iv: "bf8d61e22eefdab4f605eb9465bbee13"
server_key_version: "encryptionkey@0"
I20260810 15:01:54.495023 30731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:54.511976 30731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:54.513949 30731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:54.514086 30731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:54.517781 30731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37067
I20260810 15:01:54.517827 30834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37067 every 8 connection(s)
I20260810 15:01:54.518152 30731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:54.518996 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:54.521270 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58: Bootstrap starting.
I20260810 15:01:54.521893 30839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:54.522150 30839 log.cc:824] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:54.522468 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30731
I20260810 15:01:54.522599 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:54.522850 29447 external_mini_cluster.cc:1428] Setting key e2d8deacc4c8f8b01cf7d8c81ca7cd72
I20260810 15:01:54.522964 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58: No bootstrap required, opened a new log
I20260810 15:01:54.525815 30839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260810 15:01:54.525997 30839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:54.526033 30839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1888c922e3f0427b9300fb547a66ea58, State: Initialized, Role: FOLLOWER
I20260810 15:01:54.526165 30839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260810 15:01:54.526268 30839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:54.526301 30839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:54.526355 30839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:54.526970 30839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260810 15:01:54.527086 30839 leader_election.cc:302] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1888c922e3f0427b9300fb547a66ea58; no voters:
I20260810 15:01:54.527268 30839 leader_election.cc:288] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:54.527307 30847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:54.527453 30847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [term 1 LEADER]: Becoming Leader. State: Replica: 1888c922e3f0427b9300fb547a66ea58, State: Running, Role: LEADER
I20260810 15:01:54.527534 30839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:54.527556 30847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } }
I20260810 15:01:54.527912 30851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1888c922e3f0427b9300fb547a66ea58. Latest consensus state: current_term: 1 leader_uuid: "1888c922e3f0427b9300fb547a66ea58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260810 15:01:54.528007 30851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:54.527907 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1888c922e3f0427b9300fb547a66ea58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1888c922e3f0427b9300fb547a66ea58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37067 } } }
I20260810 15:01:54.528174 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:54.528280 30856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:54.528785 30856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:54.529444 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:54.523382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50599 (local address 127.0.0.1:37067)
0810 15:01:54.523585 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:54.523592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:54.523619 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:01:54.524196 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:54.524206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:54.524274 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:54.524431 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:54.524662 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:54.526055 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:54.526964 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:54.527571 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:54.528124 (+ 553us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:54.528580 (+ 456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:54.528585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:54.528585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:54.528830 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:54.528872 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:54.528888 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:54.529228 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:54.529267 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 15:01:54.529291 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260810 15:01:54.530453 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:01:54.534514 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:54.531838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60819 (local address 127.0.0.1:37067)
0810 15:01:54.531877 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:54.531878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:54.531933 (+ 55us) server_negotiation.cc:407] Connection header received
0810 15:01:54.531996 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:54.531997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:54.532006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:54.532049 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:54.532174 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:54.532632 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:54.533299 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:54.533499 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:54.533551 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:54.534290 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:54.534291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:54.534291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:54.534295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:54.534308 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:54.534311 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:54.534360 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:54.534429 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:01:54.534438 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:01:54.537884 30856 catalog_manager.cc:1347] Generated new cluster ID: 8cf760832a3a405fba712e2db69199cf
I20260810 15:01:54.537937 30856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:54.550089 30856 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:54.550843 30856 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:54.557300 30856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1888c922e3f0427b9300fb547a66ea58: Generated new TSK 0
I20260810 15:01:54.557507 30856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:54.560586 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30731
W20260810 15:01:54.576905 30705 connection.cc:535] client connection to 127.0.0.1:37067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37067): Connection reset by peer (error 104)
2026-08-10T15:01:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:54 dist-test-slave-cvwc krb5kdc[30886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:54 dist-test-slave-cvwc krb5kdc[30886](info): set up 2 sockets
Aug 10 15:01:54 dist-test-slave-cvwc krb5kdc[30886](info): commencing operation
krb5kdc: starting...
W20260810 15:01:55.725347 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:55 dist-test-slave-cvwc krb5kdc[30886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:01:55.839907 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35275
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:55.922456 30903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:55.922675 30903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:55.922706 30903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:55.923823 30903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:55.923864 30903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:55.923892 30903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:55.923907 30903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:55.923934 30903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:55.925405 30903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35275
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:55.925637 30903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:55.925872 30903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:01:55.930608 30903 server_base.cc:1034] running on GCE node
W20260810 15:01:55.930622 30912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:55.930570 30915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:55.930578 30917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:55.930979 30903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:55.931262 30903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:55.932413 30903 hybrid_clock.cc:648] HybridClock initialized: now 1786374115932391 us; error 32 us; skew 500 ppm
Aug 10 15:01:55 dist-test-slave-cvwc krb5kdc[30886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:55.938118 30903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:01:55.938555 30903 webserver.cc:460] Webserver started at http://127.0.0.1:33831/ using document root <none> and password file <none>
I20260810 15:01:55.938776 30903 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:55.938827 30903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:55.938961 30903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:55.940066 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "c2e66bc060ba46fe928c4bad11081ae1"
format_stamp: "Formatted at 2026-08-10 15:01:55 on dist-test-slave-cvwc"
server_key: "577da397663d865decb0ee8502006601"
server_key_iv: "14f35167b25a4cba4a8172aa888a7664"
server_key_version: "encryptionkey@0"
I20260810 15:01:55.940414 30903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "c2e66bc060ba46fe928c4bad11081ae1"
format_stamp: "Formatted at 2026-08-10 15:01:55 on dist-test-slave-cvwc"
server_key: "577da397663d865decb0ee8502006601"
server_key_iv: "14f35167b25a4cba4a8172aa888a7664"
server_key_version: "encryptionkey@0"
I20260810 15:01:55.941757 30903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:01:55.942509 30927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:55.942668 30903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:55.942756 30903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "c2e66bc060ba46fe928c4bad11081ae1"
format_stamp: "Formatted at 2026-08-10 15:01:55 on dist-test-slave-cvwc"
server_key: "577da397663d865decb0ee8502006601"
server_key_iv: "14f35167b25a4cba4a8172aa888a7664"
server_key_version: "encryptionkey@0"
I20260810 15:01:55.942826 30903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:55.946185 30903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:55.948197 30903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:55.948324 30903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:55.952061 30903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35275
I20260810 15:01:55.952109 31012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35275 every 8 connection(s)
I20260810 15:01:55.952410 30903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:55.953505 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:55.954770 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 30903
I20260810 15:01:55.954900 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:55.955070 29447 external_mini_cluster.cc:1428] Setting key 7d5789bd4c17ac77c69ac4af282a4c2b
I20260810 15:01:55.956122 31013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1: Bootstrap starting.
I20260810 15:01:55.956817 31013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:55.957129 31013 log.cc:824] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:55.957531 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:55.955565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38375 (local address 127.0.0.1:35275)
0810 15:01:55.955766 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:55.955776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:55.955808 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:01:55.956315 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:55.956324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:55.956357 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:55.956513 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:55.957041 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:55.957045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:55.957045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:55.957050 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:55.957091 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:55.957105 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:55.957196 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:55.957227 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 15:01:55.957380 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260810 15:01:55.957877 31013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1: No bootstrap required, opened a new log
I20260810 15:01:55.958426 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:01:55.959928 31013 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260810 15:01:55.960060 31013 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:55.960093 31013 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e66bc060ba46fe928c4bad11081ae1, State: Initialized, Role: FOLLOWER
I20260810 15:01:55.960214 31013 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260810 15:01:55.960306 31013 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:55.960350 31013 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:55.960397 31013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:55.960711 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:55.959652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50621 (local address 127.0.0.1:35275)
0810 15:01:55.959686 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:55.959687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:55.959883 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:01:55.959941 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:55.959942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:55.959947 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:55.959992 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:55.960534 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:55.960535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:55.960535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:55.960540 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:55.960552 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:55.960557 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:55.960601 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:55.960640 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 15:01:55.960652 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:01:55.960997 31013 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260810 15:01:55.961103 31013 leader_election.cc:302] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e66bc060ba46fe928c4bad11081ae1; no voters:
I20260810 15:01:55.961259 31013 leader_election.cc:288] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:55.961292 31031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:55.961416 31031 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [term 1 LEADER]: Becoming Leader. State: Replica: c2e66bc060ba46fe928c4bad11081ae1, State: Running, Role: LEADER
I20260810 15:01:55.961495 31013 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:55.961544 31031 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } }
I20260810 15:01:55.961943 31033 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e66bc060ba46fe928c4bad11081ae1. Latest consensus state: current_term: 1 leader_uuid: "c2e66bc060ba46fe928c4bad11081ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } } }
I20260810 15:01:55.961939 31032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e66bc060ba46fe928c4bad11081ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e66bc060ba46fe928c4bad11081ae1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35275 } } }
I20260810 15:01:55.962059 31032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:55.962260 31033 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:55.962602 31041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:55.963053 31041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:55.964440 31041 catalog_manager.cc:1347] Generated new cluster ID: 09f8159f44b44696ab886e84f41f24f7
I20260810 15:01:55.964493 31041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:55.998345 31041 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:55.998553 31041 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:56.005038 31041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e66bc060ba46fe928c4bad11081ae1: Generated new TSK 0
I20260810 15:01:56.005218 31041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:56.011308 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 30903
2026-08-10T15:01:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:56 dist-test-slave-cvwc krb5kdc[31067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:56 dist-test-slave-cvwc krb5kdc[31067](info): set up 2 sockets
Aug 10 15:01:56 dist-test-slave-cvwc krb5kdc[31067](info): commencing operation
krb5kdc: starting...
W20260810 15:01:57.175845 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:01:57.292531 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:57.372889 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:57.373085 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:57.373106 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:57.374199 31090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:57.374234 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:57.374246 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:57.374258 31090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:57.374274 31090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:57.375667 31090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:57.375865 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:57.376044 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:57.380722 31097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:57.380815 31090 server_base.cc:1034] running on GCE node
W20260810 15:01:57.380755 31096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:57.380755 31099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:57.381165 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:57.381444 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:57.382583 31090 hybrid_clock.cc:648] HybridClock initialized: now 1786374117382571 us; error 28 us; skew 500 ppm
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:57.384975 31090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:01:57.385380 31090 webserver.cc:460] Webserver started at http://127.0.0.1:38197/ using document root <none> and password file <none>
I20260810 15:01:57.385589 31090 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:57.385637 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:57.385789 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:57.386812 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "69142d6aedbd47b68b53b0d63348f5ba"
format_stamp: "Formatted at 2026-08-10 15:01:57 on dist-test-slave-cvwc"
server_key: "7e4cf3ae37d26960e3818b60a2ebdacf"
server_key_iv: "98e44a24ca149ad92d496522b8a55ca7"
server_key_version: "encryptionkey@0"
I20260810 15:01:57.387151 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "69142d6aedbd47b68b53b0d63348f5ba"
format_stamp: "Formatted at 2026-08-10 15:01:57 on dist-test-slave-cvwc"
server_key: "7e4cf3ae37d26960e3818b60a2ebdacf"
server_key_iv: "98e44a24ca149ad92d496522b8a55ca7"
server_key_version: "encryptionkey@0"
I20260810 15:01:57.388386 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:01:57.389009 31108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:57.389171 31090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:01:57.389274 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "69142d6aedbd47b68b53b0d63348f5ba"
format_stamp: "Formatted at 2026-08-10 15:01:57 on dist-test-slave-cvwc"
server_key: "7e4cf3ae37d26960e3818b60a2ebdacf"
server_key_iv: "98e44a24ca149ad92d496522b8a55ca7"
server_key_version: "encryptionkey@0"
I20260810 15:01:57.389335 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:57.405224 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:57.407424 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:57.407568 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:57.411567 31090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34035
I20260810 15:01:57.411599 31183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34035 every 8 connection(s)
I20260810 15:01:57.411986 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:57.412875 31184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:57.415162 31184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba: Bootstrap starting.
I20260810 15:01:57.415719 31184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:57.415978 31184 log.cc:824] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:57.416674 31184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba: No bootstrap required, opened a new log
I20260810 15:01:57.417598 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31090
I20260810 15:01:57.417743 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:57.417939 29447 external_mini_cluster.cc:1428] Setting key 5466d9841df8434ac9aba14a88c1f0e5
I20260810 15:01:57.418675 31184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260810 15:01:57.418823 31184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:57.418855 31184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69142d6aedbd47b68b53b0d63348f5ba, State: Initialized, Role: FOLLOWER
I20260810 15:01:57.418983 31184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260810 15:01:57.419071 31184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:57.419103 31184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:57.419195 31184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:57.419836 31184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260810 15:01:57.419926 31184 leader_election.cc:302] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69142d6aedbd47b68b53b0d63348f5ba; no voters:
I20260810 15:01:57.420074 31184 leader_election.cc:288] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:57.420120 31196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:57.420231 31196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [term 1 LEADER]: Becoming Leader. State: Replica: 69142d6aedbd47b68b53b0d63348f5ba, State: Running, Role: LEADER
I20260810 15:01:57.420336 31184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:57.420377 31196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } }
I20260810 15:01:57.420728 31196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69142d6aedbd47b68b53b0d63348f5ba. Latest consensus state: current_term: 1 leader_uuid: "69142d6aedbd47b68b53b0d63348f5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } } }
I20260810 15:01:57.421056 31196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:57.420984 31199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69142d6aedbd47b68b53b0d63348f5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69142d6aedbd47b68b53b0d63348f5ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34035 } } }
I20260810 15:01:57.421206 31199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:57.421576 31213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:57.422127 31213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:57.423627 31213 catalog_manager.cc:1347] Generated new cluster ID: d99949aed9c94a6c91cb98829b61b765
I20260810 15:01:57.423679 31213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:01:57.427876 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:57.418461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34123 (local address 127.0.0.1:34035)
0810 15:01:57.418663 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:57.418673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:57.418699 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:01:57.418796 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:57.418801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:57.418841 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:57.418984 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:57.419709 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:57.420918 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:57.421995 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:57.422455 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:57.422554 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:57.424924 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:57.424928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:57.424929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:57.424940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:57.426406 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:57.426712 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:57.426715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:57.426716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:57.426816 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:57.426992 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:57.426993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:57.426993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:57.427214 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:57.427383 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:57.427450 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 15:01:57.427584 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260810 15:01:57.428740 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:01:57.434680 31213 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:57.435333 31213 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:57.439556 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:57.433982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47895 (local address 127.0.0.1:34035)
0810 15:01:57.434018 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:57.434019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:57.434140 (+ 121us) server_negotiation.cc:407] Connection header received
0810 15:01:57.434191 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:57.434192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:57.434203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:57.434246 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:57.434862 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:57.435324 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:57.436010 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:57.436208 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:57.436260 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:57.437319 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:57.437320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:57.437321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:01:57.437327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:57.438598 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:57.438901 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:57.438902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:57.438902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:57.438955 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:01:57.439110 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:01:57.439110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:01:57.439111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:01:57.439214 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:57.439293 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:57.439391 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 15:01:57.439474 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:01:57.441159 31213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69142d6aedbd47b68b53b0d63348f5ba: Generated new TSK 0
I20260810 15:01:57.441327 31213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:57.446899 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31090
2026-08-10T15:01:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31241](info): set up 2 sockets
Aug 10 15:01:57 dist-test-slave-cvwc krb5kdc[31241](info): commencing operation
krb5kdc: starting...
W20260810 15:01:58.608615 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:01:58 dist-test-slave-cvwc krb5kdc[31241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:01:58.727833 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:01:58.814150 31258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:01:58.814354 31258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:01:58.814379 31258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:01:58.815479 31258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:01:58.815513 31258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:01:58.815526 31258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:01:58.815536 31258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:01:58.815549 31258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:01:58.816977 31258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:01:58.817173 31258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:01:58.817346 31258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:01:58.822024 31264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:58.822021 31267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:01:58.822042 31263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:01:58.822252 31258 server_base.cc:1034] running on GCE node
I20260810 15:01:58.822510 31258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:01:58.822796 31258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:01:58.823937 31258 hybrid_clock.cc:648] HybridClock initialized: now 1786374118823924 us; error 30 us; skew 500 ppm
Aug 10 15:01:58 dist-test-slave-cvwc krb5kdc[31241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:01:58.826407 31258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:01:58.826807 31258 webserver.cc:460] Webserver started at http://127.0.0.1:44645/ using document root <none> and password file <none>
I20260810 15:01:58.827044 31258 fs_manager.cc:359] Metadata directory not provided
I20260810 15:01:58.827092 31258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:01:58.827220 31258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:01:58.828250 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "cb87151049484c969da194d36e8fbf62"
format_stamp: "Formatted at 2026-08-10 15:01:58 on dist-test-slave-cvwc"
server_key: "358d7ea5265257db905d43467d1434f2"
server_key_iv: "c71f76e08bcdb3ef27a3e9f667ee0f1e"
server_key_version: "encryptionkey@0"
I20260810 15:01:58.828585 31258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "cb87151049484c969da194d36e8fbf62"
format_stamp: "Formatted at 2026-08-10 15:01:58 on dist-test-slave-cvwc"
server_key: "358d7ea5265257db905d43467d1434f2"
server_key_iv: "c71f76e08bcdb3ef27a3e9f667ee0f1e"
server_key_version: "encryptionkey@0"
I20260810 15:01:58.829905 31258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:01:58.830595 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:01:58.830749 31258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:01:58.830832 31258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "cb87151049484c969da194d36e8fbf62"
format_stamp: "Formatted at 2026-08-10 15:01:58 on dist-test-slave-cvwc"
server_key: "358d7ea5265257db905d43467d1434f2"
server_key_iv: "c71f76e08bcdb3ef27a3e9f667ee0f1e"
server_key_version: "encryptionkey@0"
I20260810 15:01:58.830909 31258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:01:58.851650 31258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:01:58.853377 31258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:01:58.853514 31258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:01:58.857353 31258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42449
I20260810 15:01:58.857388 31356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42449 every 8 connection(s)
I20260810 15:01:58.857735 31258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:01:58.858608 31358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:01:58.860816 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: Bootstrap starting.
I20260810 15:01:58.861357 31358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: Neither blocks nor log segments found. Creating new log.
I20260810 15:01:58.861589 31358 log.cc:824] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: Log is configured to *not* fsync() on all Append() calls
I20260810 15:01:58.862326 31358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: No bootstrap required, opened a new log
I20260810 15:01:58.863561 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31258
I20260810 15:01:58.863682 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:01:58.863847 29447 external_mini_cluster.cc:1428] Setting key 1fa7548f0c787df1ba77696c573e1ed8
I20260810 15:01:58.863971 31358 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260810 15:01:58.864154 31358 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:01:58.864202 31358 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb87151049484c969da194d36e8fbf62, State: Initialized, Role: FOLLOWER
I20260810 15:01:58.864351 31358 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260810 15:01:58.864444 31358 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:01:58.864476 31358 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:01:58.864542 31358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:01:58.865628 31358 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260810 15:01:58.865768 31358 leader_election.cc:302] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb87151049484c969da194d36e8fbf62; no voters:
I20260810 15:01:58.865932 31358 leader_election.cc:288] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:01:58.865967 31366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:01:58.866120 31366 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [term 1 LEADER]: Becoming Leader. State: Replica: cb87151049484c969da194d36e8fbf62, State: Running, Role: LEADER
I20260810 15:01:58.866161 31358 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:01:58.866266 31366 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } }
I20260810 15:01:58.866643 31366 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb87151049484c969da194d36e8fbf62. Latest consensus state: current_term: 1 leader_uuid: "cb87151049484c969da194d36e8fbf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } } }
I20260810 15:01:58.866770 31366 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [sys.catalog]: This master's current role is: LEADER
I20260810 15:01:58.867720 31368 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb87151049484c969da194d36e8fbf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb87151049484c969da194d36e8fbf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42449 } } }
I20260810 15:01:58.867808 31368 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62 [sys.catalog]: This master's current role is: LEADER
W20260810 15:01:58.868824 31393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:01:58.868875 31393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:01:58.868935 31380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:01:58.869109 31380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:01:58.870205 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:58.864423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55079 (local address 127.0.0.1:42449)
0810 15:01:58.864630 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:58.864640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:58.864672 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:01:58.864805 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:58.864813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:58.864896 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:58.865051 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:58.865211 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:58.866558 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:58.867392 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:58.867994 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:58.868110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:58.869332 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:58.869335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:58.869336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:58.869552 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:58.869590 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:58.869604 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:58.869676 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:58.869918 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 15:01:58.869990 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260810 15:01:58.870656 31380 catalog_manager.cc:1347] Generated new cluster ID: a145cb8195ad42a98dd156960351f7db
I20260810 15:01:58.870708 31380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:01:58.871114 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:01:58.875926 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:01:58.872563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47043 (local address 127.0.0.1:42449)
0810 15:01:58.872620 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:01:58.872622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:01:58.872803 (+ 181us) server_negotiation.cc:407] Connection header received
0810 15:01:58.872865 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:01:58.872865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:01:58.872876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:01:58.872927 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:01:58.873047 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:58.873734 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:58.874690 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:01:58.874977 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:01:58.875009 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:01:58.875713 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:01:58.875714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:01:58.875714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:01:58.875718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:01:58.875730 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:01:58.875734 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:01:58.875756 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0810 15:01:58.875836 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:01:58.875857 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:01:58.885413 31380 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:01:58.886147 31380 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:01:58.892007 31380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb87151049484c969da194d36e8fbf62: Generated new TSK 0
I20260810 15:01:58.892196 31380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:01:58.897325 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31258
2026-08-10T15:01:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:01:58 dist-test-slave-cvwc krb5kdc[31409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:01:58 dist-test-slave-cvwc krb5kdc[31409](info): set up 2 sockets
Aug 10 15:01:58 dist-test-slave-cvwc krb5kdc[31409](info): commencing operation
krb5kdc: starting...
W20260810 15:02:00.079828 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:00 dist-test-slave-cvwc krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:00.202203 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:00.284365 31428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:00.284583 31428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:00.284614 31428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:00.285782 31428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:00.285830 31428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:00.285849 31428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:00.285866 31428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:00.285887 31428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:00.287500 31428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:00.287772 31428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:00.287986 31428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:00.292649 31447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:00.292660 31439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:00.292660 31440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:00.292877 31428 server_base.cc:1034] running on GCE node
I20260810 15:02:00.293198 31428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:00.293502 31428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:00.294651 31428 hybrid_clock.cc:648] HybridClock initialized: now 1786374120294634 us; error 28 us; skew 500 ppm
Aug 10 15:02:00 dist-test-slave-cvwc krb5kdc[31409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:00.297186 31428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:00.297611 31428 webserver.cc:460] Webserver started at http://127.0.0.1:41457/ using document root <none> and password file <none>
I20260810 15:02:00.297859 31428 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:00.297923 31428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:00.298056 31428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:00.299141 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "a1155663c5f140babbb7e23949fe403c"
format_stamp: "Formatted at 2026-08-10 15:02:00 on dist-test-slave-cvwc"
server_key: "669ff3d92fca30029c6cbcd61372b446"
server_key_iv: "c624ea5f813b29ec38f598622eca3284"
server_key_version: "encryptionkey@0"
I20260810 15:02:00.299489 31428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "a1155663c5f140babbb7e23949fe403c"
format_stamp: "Formatted at 2026-08-10 15:02:00 on dist-test-slave-cvwc"
server_key: "669ff3d92fca30029c6cbcd61372b446"
server_key_iv: "c624ea5f813b29ec38f598622eca3284"
server_key_version: "encryptionkey@0"
I20260810 15:02:00.300802 31428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:00.301532 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:00.301693 31428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:00.301802 31428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "a1155663c5f140babbb7e23949fe403c"
format_stamp: "Formatted at 2026-08-10 15:02:00 on dist-test-slave-cvwc"
server_key: "669ff3d92fca30029c6cbcd61372b446"
server_key_iv: "c624ea5f813b29ec38f598622eca3284"
server_key_version: "encryptionkey@0"
I20260810 15:02:00.301870 31428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:00.305051 31428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:00.307062 31428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:00.307206 31428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:00.311021 31428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34151
I20260810 15:02:00.311059 31531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34151 every 8 connection(s)
I20260810 15:02:00.311364 31428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:00.312331 31534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:00.314639 31534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c: Bootstrap starting.
I20260810 15:02:00.315192 31534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:00.315450 31534 log.cc:824] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:00.316120 31534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c: No bootstrap required, opened a new log
I20260810 15:02:00.317495 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31428
I20260810 15:02:00.317615 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:00.317833 29447 external_mini_cluster.cc:1428] Setting key 4cb5d9f305e01a28b64696fc39589e6c
I20260810 15:02:00.317862 31534 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260810 15:02:00.318030 31534 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:00.318068 31534 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1155663c5f140babbb7e23949fe403c, State: Initialized, Role: FOLLOWER
I20260810 15:02:00.318216 31534 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260810 15:02:00.318337 31534 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:00.318410 31534 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:00.318485 31534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:00.319198 31534 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260810 15:02:00.319319 31534 leader_election.cc:302] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1155663c5f140babbb7e23949fe403c; no voters:
I20260810 15:02:00.319515 31534 leader_election.cc:288] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:00.319563 31541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:00.319741 31541 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [term 1 LEADER]: Becoming Leader. State: Replica: a1155663c5f140babbb7e23949fe403c, State: Running, Role: LEADER
I20260810 15:02:00.319802 31534 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:00.319919 31541 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } }
I20260810 15:02:00.320333 31541 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1155663c5f140babbb7e23949fe403c. Latest consensus state: current_term: 1 leader_uuid: "a1155663c5f140babbb7e23949fe403c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } } }
I20260810 15:02:00.320456 31541 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:00.320528 31544 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1155663c5f140babbb7e23949fe403c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1155663c5f140babbb7e23949fe403c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34151 } } }
I20260810 15:02:00.320659 31544 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:00.320840 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:00.318366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44463 (local address 127.0.0.1:34151)
0810 15:02:00.318662 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:00.318673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:00.318705 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:02:00.319102 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:00.319111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:00.319149 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:00.319306 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:00.320021 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:00.320026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:00.320026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:00.320030 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:00.320083 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:00.320099 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:00.320195 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:00.320314 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 15:02:00.320360 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260810 15:02:00.320701 31554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:00.321750 31554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:00.321871 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:00.323799 31554 catalog_manager.cc:1347] Generated new cluster ID: f383abbbef9a422e8953bf05365e514b
I20260810 15:02:00.323855 31554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:00.327023 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:00.323368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48383 (local address 127.0.0.1:34151)
0810 15:02:00.323403 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:00.323405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:00.326034 (+ 2629us) server_negotiation.cc:407] Connection header received
0810 15:02:00.326102 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:00.326103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:00.326123 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:00.326188 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:00.326800 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:00.326802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:00.326802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:00.326807 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:00.326822 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:00.326827 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:00.326881 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:00.326944 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 15:02:00.326957 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:02:00.330116 31554 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:00.330281 31554 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:00.334649 31554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1155663c5f140babbb7e23949fe403c: Generated new TSK 0
I20260810 15:02:00.334941 31554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:00.339530 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31428
2026-08-10T15:02:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:00 dist-test-slave-cvwc krb5kdc[31576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:00 dist-test-slave-cvwc krb5kdc[31576](info): set up 2 sockets
Aug 10 15:02:00 dist-test-slave-cvwc krb5kdc[31576](info): commencing operation
krb5kdc: starting...
W20260810 15:02:01.502247 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:01.618865 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:01.701390 31595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:01.701588 31595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:01.701609 31595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:01.702716 31595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:01.702755 31595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:01.702768 31595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:01.702778 31595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:01.702790 31595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:01.704226 31595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:01.704432 31595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:01.704599 31595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:02:01.709223 31595 server_base.cc:1034] running on GCE node
W20260810 15:02:01.709188 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:01.709188 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:01.709170 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:01.709640 31595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:01.709952 31595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:01.711086 31595 hybrid_clock.cc:648] HybridClock initialized: now 1786374121711069 us; error 34 us; skew 500 ppm
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:01.717061 31595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:01.717506 31595 webserver.cc:460] Webserver started at http://127.0.0.1:41203/ using document root <none> and password file <none>
I20260810 15:02:01.717707 31595 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:01.717793 31595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:01.717931 31595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:01.719029 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "8eb1376095db4f3aa93a3b6d0ba8f178"
format_stamp: "Formatted at 2026-08-10 15:02:01 on dist-test-slave-cvwc"
server_key: "295b20855c328bec2e988deb1f34e3fe"
server_key_iv: "ec339462d07e087e00f4397a0a21b5c2"
server_key_version: "encryptionkey@0"
I20260810 15:02:01.719398 31595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "8eb1376095db4f3aa93a3b6d0ba8f178"
format_stamp: "Formatted at 2026-08-10 15:02:01 on dist-test-slave-cvwc"
server_key: "295b20855c328bec2e988deb1f34e3fe"
server_key_iv: "ec339462d07e087e00f4397a0a21b5c2"
server_key_version: "encryptionkey@0"
I20260810 15:02:01.720693 31595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:01.721402 31622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:01.721552 31595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:01.721644 31595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "8eb1376095db4f3aa93a3b6d0ba8f178"
format_stamp: "Formatted at 2026-08-10 15:02:01 on dist-test-slave-cvwc"
server_key: "295b20855c328bec2e988deb1f34e3fe"
server_key_iv: "ec339462d07e087e00f4397a0a21b5c2"
server_key_version: "encryptionkey@0"
I20260810 15:02:01.721706 31595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:01.739859 31595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:01.741571 31595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:01.741703 31595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:01.745500 31595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44695
I20260810 15:02:01.745538 31698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44695 every 8 connection(s)
I20260810 15:02:01.745882 31595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:01.746744 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:01.748998 31700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178: Bootstrap starting.
I20260810 15:02:01.749550 31700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:01.749850 31700 log.cc:824] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:01.750514 31700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178: No bootstrap required, opened a new log
I20260810 15:02:01.752056 31700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 15:02:01.752198 31700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:01.752219 31700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb1376095db4f3aa93a3b6d0ba8f178, State: Initialized, Role: FOLLOWER
I20260810 15:02:01.752331 31700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 15:02:01.752415 31700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:01.752445 31700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:01.752480 31700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:01.752969 31700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 15:02:01.753057 31700 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb1376095db4f3aa93a3b6d0ba8f178; no voters:
I20260810 15:02:01.753185 31700 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:01.753286 31703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:01.753410 31700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:01.753484 31703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb1376095db4f3aa93a3b6d0ba8f178, State: Running, Role: LEADER
I20260810 15:02:01.753575 31703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } }
I20260810 15:02:01.753839 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31595
I20260810 15:02:01.753973 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:01.754197 29447 external_mini_cluster.cc:1428] Setting key 03710aaf7618a1c604b2a7c1351ec9d4
I20260810 15:02:01.754205 31704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260810 15:02:01.754287 31704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:01.754201 31705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb1376095db4f3aa93a3b6d0ba8f178. Latest consensus state: current_term: 1 leader_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb1376095db4f3aa93a3b6d0ba8f178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44695 } } }
I20260810 15:02:01.754587 31705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:01.754916 31712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:01.755486 31712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:01.756964 31712 catalog_manager.cc:1347] Generated new cluster ID: 0b8a743edbc84ae19cc506c4fd0f358b
I20260810 15:02:01.757014 31712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:02:01.763064 31712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:01.763707 31712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:01.766280 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:01.755028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:46571 (local address 127.0.0.1:44695)
0810 15:02:01.755463 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:01.755474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:01.755505 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:02:01.755627 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:01.755635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:01.755695 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:01.755850 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:01.756535 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:01.757943 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:01.758985 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:01.759632 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:01.759745 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:01.762589 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:01.762594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:01.762594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:01.762607 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:01.764610 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:01.765081 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:01.765083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:01.765084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:01.765180 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:01.765393 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:01.765393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:01.765394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:01.765648 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:01.765835 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:01.765953 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 15:02:01.766051 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":71,"threads_started":1}
I20260810 15:02:01.767215 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:01.769591 31712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb1376095db4f3aa93a3b6d0ba8f178: Generated new TSK 0
I20260810 15:02:01.769778 31712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:01.775110 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:01.768616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:55469 (local address 127.0.0.1:44695)
0810 15:02:01.768649 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:01.768651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:01.768760 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:02:01.768827 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:01.768828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:01.768836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:01.768891 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:01.769526 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:01.770411 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:01.771086 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:01.771404 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:01.771470 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:01.772525 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:01.772527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:01.772527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:01.772534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:01.774008 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:01.774387 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:01.774388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:01.774388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:01.774447 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:01.774645 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:01.774646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:01.774646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:01.774765 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:01.774871 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:01.774956 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:02:01.775019 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:02:01.777061 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31595
W20260810 15:02:01.792202 31573 connection.cc:535] client connection to 127.0.0.1:44695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44695): Connection reset by peer (error 104)
2026-08-10T15:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31749](info): set up 2 sockets
Aug 10 15:02:01 dist-test-slave-cvwc krb5kdc[31749](info): commencing operation
krb5kdc: starting...
W20260810 15:02:02.936203 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:03 dist-test-slave-cvwc krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:03.048605 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:03.128901 31765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:03.129101 31765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:03.129132 31765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:03.130260 31765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:03.130300 31765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:03.130318 31765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:03.130334 31765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:03.130349 31765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:03.131779 31765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:03.132021 31765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:03.132226 31765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:03.136883 31775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:03.136873 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:03.136925 31765 server_base.cc:1034] running on GCE node
W20260810 15:02:03.136875 31774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:03.137315 31765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:03.137589 31765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:03.138727 31765 hybrid_clock.cc:648] HybridClock initialized: now 1786374123138710 us; error 29 us; skew 500 ppm
Aug 10 15:02:03 dist-test-slave-cvwc krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:03.141165 31765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:03.141564 31765 webserver.cc:460] Webserver started at http://127.0.0.1:39829/ using document root <none> and password file <none>
I20260810 15:02:03.141820 31765 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:03.141872 31765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:03.141997 31765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:03.143086 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "d1dbb54e66124acb9ab786d38dbbac5d"
format_stamp: "Formatted at 2026-08-10 15:02:03 on dist-test-slave-cvwc"
server_key: "b9c3fa6817ee83e00a39081b5caaa33e"
server_key_iv: "d1e6fc130d08bd08db0b0705cfabc1df"
server_key_version: "encryptionkey@0"
I20260810 15:02:03.143422 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "d1dbb54e66124acb9ab786d38dbbac5d"
format_stamp: "Formatted at 2026-08-10 15:02:03 on dist-test-slave-cvwc"
server_key: "b9c3fa6817ee83e00a39081b5caaa33e"
server_key_iv: "d1e6fc130d08bd08db0b0705cfabc1df"
server_key_version: "encryptionkey@0"
I20260810 15:02:03.144641 31765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:02:03.145328 31791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:03.145493 31765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:02:03.145574 31765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "d1dbb54e66124acb9ab786d38dbbac5d"
format_stamp: "Formatted at 2026-08-10 15:02:03 on dist-test-slave-cvwc"
server_key: "b9c3fa6817ee83e00a39081b5caaa33e"
server_key_iv: "d1e6fc130d08bd08db0b0705cfabc1df"
server_key_version: "encryptionkey@0"
I20260810 15:02:03.145637 31765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:03.170862 31765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:03.173178 31765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:03.173327 31765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:03.177310 31765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35179
I20260810 15:02:03.177392 31865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35179 every 8 connection(s)
I20260810 15:02:03.177731 31765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:03.178660 31866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:03.181082 31866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d: Bootstrap starting.
I20260810 15:02:03.181671 31866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:03.181993 31866 log.cc:824] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:03.182693 31866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d: No bootstrap required, opened a new log
I20260810 15:02:03.183395 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31765
I20260810 15:02:03.183504 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:03.183662 29447 external_mini_cluster.cc:1428] Setting key 93e9d0423dc4a9ca2013223176808914
I20260810 15:02:03.184581 31866 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260810 15:02:03.184736 31866 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:03.184773 31866 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1dbb54e66124acb9ab786d38dbbac5d, State: Initialized, Role: FOLLOWER
I20260810 15:02:03.184926 31866 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260810 15:02:03.185011 31866 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:03.185046 31866 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:03.185101 31866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:03.185643 31866 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260810 15:02:03.185776 31866 leader_election.cc:302] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1dbb54e66124acb9ab786d38dbbac5d; no voters:
I20260810 15:02:03.185953 31866 leader_election.cc:288] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:03.185994 31871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:03.186156 31871 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [term 1 LEADER]: Becoming Leader. State: Replica: d1dbb54e66124acb9ab786d38dbbac5d, State: Running, Role: LEADER
I20260810 15:02:03.186225 31866 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:03.186322 31871 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260810 15:02:03.186964 31875 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } } }
I20260810 15:02:03.187196 31875 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:03.187134 31874 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1dbb54e66124acb9ab786d38dbbac5d. Latest consensus state: current_term: 1 leader_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1dbb54e66124acb9ab786d38dbbac5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } } }
I20260810 15:02:03.187441 31874 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:03.187700 31884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:03.188411 31884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:03.189918 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:03.184159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:33413 (local address 127.0.0.1:35179)
0810 15:02:03.184366 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:03.184374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:03.184398 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:02:03.184540 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:03.184545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:03.184597 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:03.184712 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:03.184919 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:03.186581 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:03.187708 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:03.188224 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:03.188334 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:03.189114 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:03.189119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:03.189119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:03.189342 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:03.189379 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:03.189395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:03.189476 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:03.189641 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 15:02:03.189672 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260810 15:02:03.190124 31884 catalog_manager.cc:1347] Generated new cluster ID: 2c66156edbf1495290f96683a3a495cf
I20260810 15:02:03.190176 31884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:03.190902 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:03.195688 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:03.192293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:57105 (local address 127.0.0.1:35179)
0810 15:02:03.192328 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:03.192329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:03.192456 (+ 127us) server_negotiation.cc:407] Connection header received
0810 15:02:03.192507 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:03.192508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:03.192518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:03.192594 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:03.192725 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:03.193418 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:03.194396 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:03.194713 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:03.194786 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:03.195447 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:03.195448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:03.195448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:03.195453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:03.195467 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:03.195472 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:03.195525 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:03.195604 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:02:03.195617 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:02:03.201934 31884 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:03.202606 31884 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:03.210472 31884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1dbb54e66124acb9ab786d38dbbac5d: Generated new TSK 0
I20260810 15:02:03.210647 31884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:03.216714 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31765
W20260810 15:02:03.232005 31744 connection.cc:535] client connection to 127.0.0.1:35179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35179): Connection reset by peer (error 104)
2026-08-10T15:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:03 dist-test-slave-cvwc krb5kdc[31917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:03 dist-test-slave-cvwc krb5kdc[31917](info): set up 2 sockets
Aug 10 15:02:03 dist-test-slave-cvwc krb5kdc[31917](info): commencing operation
krb5kdc: starting...
W20260810 15:02:04.383298 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:04 dist-test-slave-cvwc krb5kdc[31917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:04.496177 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:04.576718 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:04.576927 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:04.576959 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:04.578136 31941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:04.578177 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:04.578195 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:04.578212 31941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:04.578226 31941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:04.579761 31941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:04.580013 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:04.580222 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:04.585105 31949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:04.585135 31947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:04.585110 31951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:04.585134 31941 server_base.cc:1034] running on GCE node
I20260810 15:02:04.585577 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:04.585876 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:04.587038 31941 hybrid_clock.cc:648] HybridClock initialized: now 1786374124587006 us; error 50 us; skew 500 ppm
Aug 10 15:02:04 dist-test-slave-cvwc krb5kdc[31917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:04.589489 31941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:04.589964 31941 webserver.cc:460] Webserver started at http://127.0.0.1:34901/ using document root <none> and password file <none>
I20260810 15:02:04.590188 31941 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:04.590237 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:04.590365 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:04.591435 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "ca7373a390064696a137d9ab4393b644"
format_stamp: "Formatted at 2026-08-10 15:02:04 on dist-test-slave-cvwc"
server_key: "72235f221f3fa1ead6f40d5d2eda271d"
server_key_iv: "9484160ce1ed25b5a9500f17a14e510c"
server_key_version: "encryptionkey@0"
I20260810 15:02:04.591773 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "ca7373a390064696a137d9ab4393b644"
format_stamp: "Formatted at 2026-08-10 15:02:04 on dist-test-slave-cvwc"
server_key: "72235f221f3fa1ead6f40d5d2eda271d"
server_key_iv: "9484160ce1ed25b5a9500f17a14e510c"
server_key_version: "encryptionkey@0"
I20260810 15:02:04.593072 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:02:04.593852 31966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:04.594064 31941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:02:04.594159 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "ca7373a390064696a137d9ab4393b644"
format_stamp: "Formatted at 2026-08-10 15:02:04 on dist-test-slave-cvwc"
server_key: "72235f221f3fa1ead6f40d5d2eda271d"
server_key_iv: "9484160ce1ed25b5a9500f17a14e510c"
server_key_version: "encryptionkey@0"
I20260810 15:02:04.594233 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:04.597527 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:04.599584 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:04.599720 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:04.603539 31941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41393
I20260810 15:02:04.603574 32062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41393 every 8 connection(s)
I20260810 15:02:04.603885 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:04.604841 32063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:04.607111 32063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644: Bootstrap starting.
I20260810 15:02:04.607652 32063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:04.607887 32063 log.cc:824] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:04.608517 32063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644: No bootstrap required, opened a new log
I20260810 15:02:04.610074 32063 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } }
I20260810 15:02:04.610212 32063 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:04.610234 32063 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7373a390064696a137d9ab4393b644, State: Initialized, Role: FOLLOWER
I20260810 15:02:04.610335 32063 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } }
I20260810 15:02:04.610380 32063 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:04.610399 32063 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:04.610428 32063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:04.610914 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 31941
I20260810 15:02:04.610972 32063 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } }
I20260810 15:02:04.611045 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:04.611079 32063 leader_election.cc:302] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7373a390064696a137d9ab4393b644; no voters:
I20260810 15:02:04.611238 32063 leader_election.cc:288] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:04.611253 29447 external_mini_cluster.cc:1428] Setting key 5809750835158bc0fcde277704f00d37
I20260810 15:02:04.611279 32069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:04.611423 32069 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [term 1 LEADER]: Becoming Leader. State: Replica: ca7373a390064696a137d9ab4393b644, State: Running, Role: LEADER
I20260810 15:02:04.611501 32063 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:04.611555 32069 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } }
I20260810 15:02:04.611989 32073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7373a390064696a137d9ab4393b644. Latest consensus state: current_term: 1 leader_uuid: "ca7373a390064696a137d9ab4393b644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } } }
I20260810 15:02:04.612066 32073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:04.612205 32070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7373a390064696a137d9ab4393b644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7373a390064696a137d9ab4393b644" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41393 } } }
I20260810 15:02:04.612255 32070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:04.612569 32077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:04.613080 32077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:04.614006 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:04.611720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:34511 (local address 127.0.0.1:41393)
0810 15:02:04.611891 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:04.611901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:04.612127 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:02:04.612259 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:04.612268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:04.612304 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:04.612465 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:04.613476 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:04.613480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:04.613481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:04.613485 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:04.613532 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:04.613548 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:04.613606 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:04.613771 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 15:02:04.613822 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260810 15:02:04.614568 32077 catalog_manager.cc:1347] Generated new cluster ID: a0534e5068e3429388b06e009ccd7aa1
I20260810 15:02:04.614620 32077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:04.614905 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:04.622761 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:04.618067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:42805 (local address 127.0.0.1:41393)
0810 15:02:04.618106 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:04.618108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:04.621864 (+ 3756us) server_negotiation.cc:407] Connection header received
0810 15:02:04.621927 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:04.621928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:04.621942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:04.621998 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:04.622534 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:04.622535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:04.622535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:04.622540 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:04.622555 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:04.622559 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:04.622607 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:04.622663 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:02:04.622678 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:04.632655 32077 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:04.632828 32077 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:04.636075 32077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7373a390064696a137d9ab4393b644: Generated new TSK 0
I20260810 15:02:04.636250 32077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:04.639575 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 31941
2026-08-10T15:02:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:04 dist-test-slave-cvwc krb5kdc[32115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:04 dist-test-slave-cvwc krb5kdc[32115](info): set up 2 sockets
Aug 10 15:02:04 dist-test-slave-cvwc krb5kdc[32115](info): commencing operation
krb5kdc: starting...
W20260810 15:02:05.814638 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:05 dist-test-slave-cvwc krb5kdc[32115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:05.930138 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:06.012163 32145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:06.012391 32145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:06.012434 32145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:06.013592 32145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:06.013633 32145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:06.013651 32145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:06.013667 32145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:06.013681 32145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:06.015213 32145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:06.015471 32145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:06.015688 32145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:06.020407 32153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:06.020504 32145 server_base.cc:1034] running on GCE node
W20260810 15:02:06.020459 32158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:06.020464 32162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:06.020885 32145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:06.021167 32145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:06.022336 32145 hybrid_clock.cc:648] HybridClock initialized: now 1786374126022316 us; error 33 us; skew 500 ppm
Aug 10 15:02:06 dist-test-slave-cvwc krb5kdc[32115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:06.024789 32145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:06.025200 32145 webserver.cc:460] Webserver started at http://127.0.0.1:45709/ using document root <none> and password file <none>
I20260810 15:02:06.025404 32145 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:06.025466 32145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:06.025647 32145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:06.026703 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "380679c9b18a445dac671e708997fc1d"
format_stamp: "Formatted at 2026-08-10 15:02:06 on dist-test-slave-cvwc"
server_key: "566e34e45c0ec5f4707fd025456ed059"
server_key_iv: "e174f73f6a0192f1eb4d6ce767b87898"
server_key_version: "encryptionkey@0"
I20260810 15:02:06.027040 32145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "380679c9b18a445dac671e708997fc1d"
format_stamp: "Formatted at 2026-08-10 15:02:06 on dist-test-slave-cvwc"
server_key: "566e34e45c0ec5f4707fd025456ed059"
server_key_iv: "e174f73f6a0192f1eb4d6ce767b87898"
server_key_version: "encryptionkey@0"
I20260810 15:02:06.028290 32145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:02:06.028946 32169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:06.029112 32145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:06.029194 32145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "380679c9b18a445dac671e708997fc1d"
format_stamp: "Formatted at 2026-08-10 15:02:06 on dist-test-slave-cvwc"
server_key: "566e34e45c0ec5f4707fd025456ed059"
server_key_iv: "e174f73f6a0192f1eb4d6ce767b87898"
server_key_version: "encryptionkey@0"
I20260810 15:02:06.029263 32145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:06.046949 32145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:06.048712 32145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:06.048847 32145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:06.052670 32145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40641
I20260810 15:02:06.052704 32251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40641 every 8 connection(s)
I20260810 15:02:06.053040 32145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:06.053964 32253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:06.055169 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32145
I20260810 15:02:06.055306 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:06.055490 29447 external_mini_cluster.cc:1428] Setting key 7c441ece7624efde5a55fa0f6f44fa73
I20260810 15:02:06.056722 32253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d: Bootstrap starting.
I20260810 15:02:06.057453 32253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:06.057813 32253 log.cc:824] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:06.058485 32253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d: No bootstrap required, opened a new log
I20260810 15:02:06.060748 32253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260810 15:02:06.060932 32253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:06.060982 32253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380679c9b18a445dac671e708997fc1d, State: Initialized, Role: FOLLOWER
I20260810 15:02:06.061120 32253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260810 15:02:06.061206 32253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:06.061244 32253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:06.061295 32253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:06.061997 32253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260810 15:02:06.062098 32253 leader_election.cc:302] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 380679c9b18a445dac671e708997fc1d; no voters:
I20260810 15:02:06.062255 32253 leader_election.cc:288] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 10 15:02:06 dist-test-slave-cvwc krb5kdc[32115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:02:06.062302 32260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:06.062512 32260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [term 1 LEADER]: Becoming Leader. State: Replica: 380679c9b18a445dac671e708997fc1d, State: Running, Role: LEADER
I20260810 15:02:06.062525 32253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:06.062670 32260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } }
I20260810 15:02:06.063050 32261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "380679c9b18a445dac671e708997fc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260810 15:02:06.063126 32261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:06.063316 32262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 380679c9b18a445dac671e708997fc1d. Latest consensus state: current_term: 1 leader_uuid: "380679c9b18a445dac671e708997fc1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "380679c9b18a445dac671e708997fc1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40641 } } }
I20260810 15:02:06.063426 32262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:06.063783 32270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:06.064508 32270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:06.065999 32270 catalog_manager.cc:1347] Generated new cluster ID: 1dddb61d73f545d5ad3ace49a7850990
I20260810 15:02:06.066045 32270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:06.066998 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:06.056014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:39465 (local address 127.0.0.1:40641)
0810 15:02:06.056163 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:06.056171 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:06.056196 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:02:06.056661 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:06.056669 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:06.056721 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:06.056861 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:06.057620 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:06.058622 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:06.059635 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:06.060027 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:06.060103 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:06.062893 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:06.062897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:06.062898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:06.062911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:06.064916 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:06.065376 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:06.065380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:06.065380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:06.065479 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:06.065753 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:06.065754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:06.065755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:06.066077 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:06.066230 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:06.066678 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 15:02:06.066781 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260810 15:02:06.067963 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:06.076977 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:06.069437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:48137 (local address 127.0.0.1:40641)
0810 15:02:06.069474 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:06.069475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:06.069626 (+ 151us) server_negotiation.cc:407] Connection header received
0810 15:02:06.069743 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:06.069744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:06.069754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:06.069784 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:06.070511 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:06.071276 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:06.072298 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:06.072625 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:06.072668 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:06.074194 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:06.074195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:06.074195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:06.074202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:06.075649 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:06.076085 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:06.076086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:06.076086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:06.076145 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:06.076368 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:06.076369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:06.076369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:06.076489 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:06.076566 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:06.076800 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 15:02:06.076867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:06.094339 32270 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:06.095091 32270 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:06.103149 32270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 380679c9b18a445dac671e708997fc1d: Generated new TSK 0
I20260810 15:02:06.103375 32270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:06.107016 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32145
W20260810 15:02:06.124993 32112 connection.cc:535] client connection to 127.0.0.1:40641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40641): Connection reset by peer (error 104)
2026-08-10T15:02:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:06 dist-test-slave-cvwc krb5kdc[32303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:06 dist-test-slave-cvwc krb5kdc[32303](info): set up 2 sockets
Aug 10 15:02:06 dist-test-slave-cvwc krb5kdc[32303](info): commencing operation
krb5kdc: starting...
W20260810 15:02:07.282912 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:07 dist-test-slave-cvwc krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:07.400382 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:07.481441 32325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:07.481640 32325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:07.481671 32325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:07.482803 32325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:07.482846 32325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:07.482864 32325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:07.482879 32325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:07.482895 32325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:07.484344 32325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:07.484576 32325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:07.484772 32325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:07.489450 32336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:07.489490 32325 server_base.cc:1034] running on GCE node
W20260810 15:02:07.489461 32334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:07.489473 32333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:07.489938 32325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:07.490226 32325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:07.491369 32325 hybrid_clock.cc:648] HybridClock initialized: now 1786374127491355 us; error 26 us; skew 500 ppm
Aug 10 15:02:07 dist-test-slave-cvwc krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:07.497084 32325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:07.497532 32325 webserver.cc:460] Webserver started at http://127.0.0.1:45603/ using document root <none> and password file <none>
I20260810 15:02:07.497756 32325 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:07.497815 32325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:07.497951 32325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:07.499010 32325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb"
format_stamp: "Formatted at 2026-08-10 15:02:07 on dist-test-slave-cvwc"
server_key: "fd66ffc34cf69ea7959ae4cb45538c03"
server_key_iv: "4dace0aa2dafab7b3de1ed0cd8c41285"
server_key_version: "encryptionkey@0"
I20260810 15:02:07.499351 32325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb"
format_stamp: "Formatted at 2026-08-10 15:02:07 on dist-test-slave-cvwc"
server_key: "fd66ffc34cf69ea7959ae4cb45538c03"
server_key_iv: "4dace0aa2dafab7b3de1ed0cd8c41285"
server_key_version: "encryptionkey@0"
I20260810 15:02:07.500584 32325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:07.501263 32348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:07.501423 32325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:07.501509 32325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb"
format_stamp: "Formatted at 2026-08-10 15:02:07 on dist-test-slave-cvwc"
server_key: "fd66ffc34cf69ea7959ae4cb45538c03"
server_key_iv: "4dace0aa2dafab7b3de1ed0cd8c41285"
server_key_version: "encryptionkey@0"
I20260810 15:02:07.501574 32325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:07.520486 32325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:07.522394 32325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:07.522531 32325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:07.526383 32325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43313
I20260810 15:02:07.526418 32429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43313 every 8 connection(s)
I20260810 15:02:07.526739 32325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:07.527631 32431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:07.529922 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb: Bootstrap starting.
I20260810 15:02:07.530457 32431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:07.530681 32431 log.cc:824] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:07.531306 32431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb: No bootstrap required, opened a new log
I20260810 15:02:07.532809 32431 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20260810 15:02:07.532945 32431 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:07.532970 32431 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9d485e1419490b9b01a1a8ed3b7bdb, State: Initialized, Role: FOLLOWER
I20260810 15:02:07.533079 32431 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20260810 15:02:07.533157 32431 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:07.533186 32431 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:07.533218 32431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:07.533668 32431 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20260810 15:02:07.533787 32431 leader_election.cc:302] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba9d485e1419490b9b01a1a8ed3b7bdb; no voters:
I20260810 15:02:07.533914 32431 leader_election.cc:288] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:07.533965 32439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:07.534127 32431 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:07.534142 32439 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [term 1 LEADER]: Becoming Leader. State: Replica: ba9d485e1419490b9b01a1a8ed3b7bdb, State: Running, Role: LEADER
I20260810 15:02:07.534307 32439 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20260810 15:02:07.534680 32440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20260810 15:02:07.534809 32443 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba9d485e1419490b9b01a1a8ed3b7bdb. Latest consensus state: current_term: 1 leader_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba9d485e1419490b9b01a1a8ed3b7bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20260810 15:02:07.534883 32443 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:07.534817 32440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:07.535293 32453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:07.535310 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32325
I20260810 15:02:07.535440 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:07.535651 29447 external_mini_cluster.cc:1428] Setting key d74cd5e966dcb48dbfb0cee16f79a629
I20260810 15:02:07.535840 32453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:07.537364 32453 catalog_manager.cc:1347] Generated new cluster ID: 354cc4a54157465f86b206b7f9896062
I20260810 15:02:07.537453 32453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:07.540614 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:07.536291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:40177 (local address 127.0.0.1:43313)
0810 15:02:07.536567 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:07.536576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:07.536602 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:02:07.536695 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:07.536701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:07.536744 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:07.536889 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:07.537120 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:07.538125 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:07.538886 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:07.539316 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:07.539417 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:07.540056 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:07.540061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:07.540061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:07.540223 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:07.540252 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:07.540264 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:07.540332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:07.540425 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 15:02:07.540452 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260810 15:02:07.541586 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:07.548405 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:07.542858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:34663 (local address 127.0.0.1:43313)
0810 15:02:07.542891 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:07.542893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:07.545886 (+ 2993us) server_negotiation.cc:407] Connection header received
0810 15:02:07.545942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:07.545942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:07.545965 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:07.546012 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:07.546132 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:07.546603 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:07.547261 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:07.547460 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:07.547512 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:07.548122 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:07.548123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:07.548124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:07.548128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:07.548142 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:07.548147 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:07.548197 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:07.548318 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 15:02:07.548330 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:07.550771 32453 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:07.551265 32453 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:07.555630 32453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba9d485e1419490b9b01a1a8ed3b7bdb: Generated new TSK 0
I20260810 15:02:07.555791 32453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:07.563025 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32325
W20260810 15:02:07.577539 32300 connection.cc:535] client connection to 127.0.0.1:43313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43313): Connection reset by peer (error 104)
2026-08-10T15:02:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:07 dist-test-slave-cvwc krb5kdc[32490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:07 dist-test-slave-cvwc krb5kdc[32490](info): set up 2 sockets
Aug 10 15:02:07 dist-test-slave-cvwc krb5kdc[32490](info): commencing operation
krb5kdc: starting...
W20260810 15:02:08.731442 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:08 dist-test-slave-cvwc krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:02:08.851821 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:08.933480 32507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:08.933683 32507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:08.933743 32507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:08.934886 32507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:08.934927 32507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:08.934947 32507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:08.934962 32507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:08.934978 32507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:08.936436 32507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:08.936679 32507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:08.936894 32507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:08.941632 32518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:08.941614 32521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:08.941596 32519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:08.941808 32507 server_base.cc:1034] running on GCE node
I20260810 15:02:08.942098 32507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:08.942368 32507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:08.943544 32507 hybrid_clock.cc:648] HybridClock initialized: now 1786374128943526 us; error 28 us; skew 500 ppm
Aug 10 15:02:08 dist-test-slave-cvwc krb5kdc[32490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:08.945966 32507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:02:08.946388 32507 webserver.cc:460] Webserver started at http://127.0.0.1:39803/ using document root <none> and password file <none>
I20260810 15:02:08.946599 32507 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:08.946646 32507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:08.946767 32507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:08.947804 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "49d5c666c8de492eacc4ef574cfbdd8c"
format_stamp: "Formatted at 2026-08-10 15:02:08 on dist-test-slave-cvwc"
server_key: "66e71bbf3731639996d3ffa530e757fc"
server_key_iv: "e8f77353b548fbf15f49fff314c28ea0"
server_key_version: "encryptionkey@0"
I20260810 15:02:08.948146 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "49d5c666c8de492eacc4ef574cfbdd8c"
format_stamp: "Formatted at 2026-08-10 15:02:08 on dist-test-slave-cvwc"
server_key: "66e71bbf3731639996d3ffa530e757fc"
server_key_iv: "e8f77353b548fbf15f49fff314c28ea0"
server_key_version: "encryptionkey@0"
I20260810 15:02:08.949390 32507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:02:08.950093 32532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:08.950237 32507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:08.950310 32507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "49d5c666c8de492eacc4ef574cfbdd8c"
format_stamp: "Formatted at 2026-08-10 15:02:08 on dist-test-slave-cvwc"
server_key: "66e71bbf3731639996d3ffa530e757fc"
server_key_iv: "e8f77353b548fbf15f49fff314c28ea0"
server_key_version: "encryptionkey@0"
I20260810 15:02:08.950380 32507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:08.953624 32507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:08.955720 32507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:08.955852 32507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:08.959676 32507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36861
I20260810 15:02:08.959708 32612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36861 every 8 connection(s)
I20260810 15:02:08.960017 32507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:08.960981 32615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:08.963387 32615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: Bootstrap starting.
I20260810 15:02:08.963939 32615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:08.964185 32615 log.cc:824] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:08.964862 32615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: No bootstrap required, opened a new log
I20260810 15:02:08.966482 32615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } }
I20260810 15:02:08.966645 32615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:08.966693 32615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d5c666c8de492eacc4ef574cfbdd8c, State: Initialized, Role: FOLLOWER
I20260810 15:02:08.966768 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32507
I20260810 15:02:08.966889 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:08.966861 32615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } }
I20260810 15:02:08.966935 32615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:08.966974 32615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:08.967022 32615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:08.967093 29447 external_mini_cluster.cc:1428] Setting key 4ccd31951d1b49b3bcf9d58f1acd7dd6
I20260810 15:02:08.967629 32615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } }
I20260810 15:02:08.967742 32615 leader_election.cc:302] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d5c666c8de492eacc4ef574cfbdd8c; no voters:
I20260810 15:02:08.967913 32615 leader_election.cc:288] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:08.967958 32624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:08.968072 32624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [term 1 LEADER]: Becoming Leader. State: Replica: 49d5c666c8de492eacc4ef574cfbdd8c, State: Running, Role: LEADER
I20260810 15:02:08.968215 32624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } }
I20260810 15:02:08.968201 32615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:08.969416 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:08.967564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:52563 (local address 127.0.0.1:36861)
0810 15:02:08.967717 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:08.967727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:08.967762 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:02:08.967896 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:08.967903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:08.967931 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:08.968103 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:08.968801 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:08.968805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:08.968805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:08.968810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:08.968858 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:08.968873 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:08.968943 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:08.969191 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 15:02:08.969237 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260810 15:02:08.969468 32627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } } }
I20260810 15:02:08.969573 32627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:08.969969 32625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d5c666c8de492eacc4ef574cfbdd8c. Latest consensus state: current_term: 1 leader_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d5c666c8de492eacc4ef574cfbdd8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36861 } } }
I20260810 15:02:08.970197 32625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:08.970341 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 15:02:08.970405 32648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:08.970484 32648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:08.970536 32645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:08.970678 32645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:08.972162 32645 catalog_manager.cc:1347] Generated new cluster ID: 6f506d6f0d2b4a61afc54fb77a8540f8
I20260810 15:02:08.972218 32645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:08.972771 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:08.971864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.198:37571 (local address 127.0.0.1:36861)
0810 15:02:08.971893 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:08.971895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:08.971931 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:02:08.972003 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:08.972004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:08.972008 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:08.972044 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:08.972576 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:08.972578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:08.972578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:08.972583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:08.972597 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:08.972600 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:08.972646 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:08.972695 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 15:02:08.972709 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:02:08.981120 32645 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:08.981279 32645 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:08.986605 32645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d5c666c8de492eacc4ef574cfbdd8c: Generated new TSK 0
I20260810 15:02:08.986785 32645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:08.988478 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32507
2026-08-10T15:02:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:09 dist-test-slave-cvwc krb5kdc[32668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:09 dist-test-slave-cvwc krb5kdc[32668](info): set up 2 sockets
Aug 10 15:02:09 dist-test-slave-cvwc krb5kdc[32668](info): commencing operation
krb5kdc: starting...
W20260810 15:02:10.158989 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[32668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:10.284384 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37863
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:37863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:10.367787 32686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:10.368009 32686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:10.368042 32686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:10.369437 32686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:10.369508 32686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:10.369534 32686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:10.369553 32686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:10.369582 32686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:10.371336 32686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:37863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:10.371558 32686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:10.371739 32686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:10.376529 32696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:10.376582 32693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:10.376545 32692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:10.376574 32686 server_base.cc:1034] running on GCE node
I20260810 15:02:10.376998 32686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:10.377281 32686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:10.378449 32686 hybrid_clock.cc:648] HybridClock initialized: now 1786374130378433 us; error 29 us; skew 500 ppm
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[32668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374130, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:10.380990 32686 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:10.381429 32686 webserver.cc:460] Webserver started at http://127.28.193.254:34663/ using document root <none> and password file <none>
I20260810 15:02:10.381645 32686 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:10.381685 32686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:10.381821 32686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:10.382863 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "03a93c594f554ec89b9a8e03995f412b"
format_stamp: "Formatted at 2026-08-10 15:02:10 on dist-test-slave-cvwc"
server_key: "6ff2763de1291a69368a19b17c9304dc"
server_key_iv: "1c865389251457555fa685e61ddc2525"
server_key_version: "encryptionkey@0"
I20260810 15:02:10.383179 32686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "03a93c594f554ec89b9a8e03995f412b"
format_stamp: "Formatted at 2026-08-10 15:02:10 on dist-test-slave-cvwc"
server_key: "6ff2763de1291a69368a19b17c9304dc"
server_key_iv: "1c865389251457555fa685e61ddc2525"
server_key_version: "encryptionkey@0"
I20260810 15:02:10.384402 32686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:02:10.385041 32704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:10.385197 32686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:02:10.385282 32686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "03a93c594f554ec89b9a8e03995f412b"
format_stamp: "Formatted at 2026-08-10 15:02:10 on dist-test-slave-cvwc"
server_key: "6ff2763de1291a69368a19b17c9304dc"
server_key_iv: "1c865389251457555fa685e61ddc2525"
server_key_version: "encryptionkey@0"
I20260810 15:02:10.385352 32686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:10.404162 32686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:10.406770 32686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:10.406936 32686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:10.415612 32686 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:37863
I20260810 15:02:10.415663 310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:37863 every 8 connection(s)
I20260810 15:02:10.415973 32686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:10.416844 312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:10.419512 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 32686
I20260810 15:02:10.419648 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:10.419633 312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: Bootstrap starting.
I20260810 15:02:10.419839 29447 external_mini_cluster.cc:1428] Setting key 45d85c17cb0330431ca0339b56b92ef6
I20260810 15:02:10.420307 312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:10.420643 312 log.cc:824] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:10.421378 312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: No bootstrap required, opened a new log
I20260810 15:02:10.424456 312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } }
I20260810 15:02:10.424667 312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:10.424710 312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a93c594f554ec89b9a8e03995f412b, State: Initialized, Role: FOLLOWER
I20260810 15:02:10.424887 312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } }
I20260810 15:02:10.424978 312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:10.425015 312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:10.425063 312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:10.425882 312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } }
I20260810 15:02:10.426014 312 leader_election.cc:302] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03a93c594f554ec89b9a8e03995f412b; no voters:
I20260810 15:02:10.426447 312 leader_election.cc:288] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:10.426455 318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:10.426620 318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [term 1 LEADER]: Becoming Leader. State: Replica: 03a93c594f554ec89b9a8e03995f412b, State: Running, Role: LEADER
I20260810 15:02:10.426715 318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } }
I20260810 15:02:10.426759 312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:10.427079 320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03a93c594f554ec89b9a8e03995f412b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } } }
I20260810 15:02:10.427165 320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [sys.catalog]: This master's current role is: LEADER
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[32668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
W20260810 15:02:10.428486 342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:10.428645 342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:10.429276 322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03a93c594f554ec89b9a8e03995f412b. Latest consensus state: current_term: 1 leader_uuid: "03a93c594f554ec89b9a8e03995f412b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03a93c594f554ec89b9a8e03995f412b" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37863 } } }
I20260810 15:02:10.429379 322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:10.429488 327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:10.429674 327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:10.431352 327 catalog_manager.cc:1347] Generated new cluster ID: 334e357b7be94e79b0804e4ec7577809
I20260810 15:02:10.431404 327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:10.436748 315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:10.420397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40721 (local address 127.28.193.254:37863)
0810 15:02:10.420592 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:10.420601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:10.421392 (+ 791us) server_negotiation.cc:407] Connection header received
0810 15:02:10.422056 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:10.422065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:10.422118 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:10.422212 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:10.422925 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:10.423810 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:10.424657 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:10.425176 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:10.425242 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:10.429515 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:10.429520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:10.429521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:10.429534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:10.434849 (+ 5315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:10.435313 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:10.435316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:10.435317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:10.435425 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:10.435657 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:10.435658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:10.435658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:10.435977 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:10.436147 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:10.436284 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 15:02:10.436383 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260810 15:02:10.437829 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:10.444856 327 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:10.446321 327 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:10.446913 315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:10.439412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59037 (local address 127.28.193.254:37863)
0810 15:02:10.439451 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:10.439453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:10.439537 (+ 84us) server_negotiation.cc:407] Connection header received
0810 15:02:10.439603 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:10.439604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:10.439613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:10.439673 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:10.440304 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:10.441070 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:10.442121 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:10.442441 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:10.442517 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:10.443917 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:10.443919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:10.443919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:10.443927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:10.445524 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:10.445978 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:10.445979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:10.445980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:10.446039 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:10.446255 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:10.446255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:10.446255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:10.446386 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:10.446464 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:10.446756 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 15:02:10.446824 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:02:10.451500 327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03a93c594f554ec89b9a8e03995f412b: Generated new TSK 0
I20260810 15:02:10.451676 327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:10.453824 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 32686
W20260810 15:02:10.466934 32664 connection.cc:535] client connection to 127.28.193.254:37863 recv error: Network error: failed to read from TLS socket (remote: 127.28.193.254:37863): Connection reset by peer (error 104)
2026-08-10T15:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[353](info): set up 2 sockets
Aug 10 15:02:10 dist-test-slave-cvwc krb5kdc[353](info): commencing operation
krb5kdc: starting...
W20260810 15:02:11.625746 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:11 dist-test-slave-cvwc krb5kdc[353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:11.744632 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35699
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:11.826510 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:11.826717 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:11.826748 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:11.827879 373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:11.827920 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:11.827939 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:11.827952 373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:11.827968 373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:11.829427 373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:35699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:11.829664 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:11.829898 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:11.834395 384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:11.834416 383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:11.834411 387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:11.834704 373 server_base.cc:1034] running on GCE node
I20260810 15:02:11.834945 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:11.835212 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:11.836361 373 hybrid_clock.cc:648] HybridClock initialized: now 1786374131836345 us; error 28 us; skew 500 ppm
Aug 10 15:02:11 dist-test-slave-cvwc krb5kdc[353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374131, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:11.838836 373 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:11.839294 373 webserver.cc:460] Webserver started at http://127.28.193.254:33445/ using document root <none> and password file <none>
I20260810 15:02:11.839501 373 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:11.839553 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:11.839689 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:11.840744 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "2455b36a5af74869a1f30f567bc5866e"
format_stamp: "Formatted at 2026-08-10 15:02:11 on dist-test-slave-cvwc"
server_key: "bff17a877943cc2c3cd6d08ee3db9de8"
server_key_iv: "2ab19fae1f66e57310d567e18e0b4e26"
server_key_version: "encryptionkey@0"
I20260810 15:02:11.841106 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "2455b36a5af74869a1f30f567bc5866e"
format_stamp: "Formatted at 2026-08-10 15:02:11 on dist-test-slave-cvwc"
server_key: "bff17a877943cc2c3cd6d08ee3db9de8"
server_key_iv: "2ab19fae1f66e57310d567e18e0b4e26"
server_key_version: "encryptionkey@0"
I20260810 15:02:11.842521 373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:02:11.843232 401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:11.843385 373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:11.843469 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "2455b36a5af74869a1f30f567bc5866e"
format_stamp: "Formatted at 2026-08-10 15:02:11 on dist-test-slave-cvwc"
server_key: "bff17a877943cc2c3cd6d08ee3db9de8"
server_key_iv: "2ab19fae1f66e57310d567e18e0b4e26"
server_key_version: "encryptionkey@0"
I20260810 15:02:11.843531 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:11.866196 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:11.869010 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:11.869190 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:11.875094 373 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:35699
I20260810 15:02:11.875124 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:35699 every 8 connection(s)
I20260810 15:02:11.875527 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:11.877908 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:11.880215 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 373
I20260810 15:02:11.880342 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:11.880519 29447 external_mini_cluster.cc:1428] Setting key 95db50ad5369e60616fcfaa4c9f1b7c2
I20260810 15:02:11.880589 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e: Bootstrap starting.
I20260810 15:02:11.881340 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:11.881613 477 log.cc:824] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:11.882275 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e: No bootstrap required, opened a new log
I20260810 15:02:11.884059 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } }
I20260810 15:02:11.884192 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:11.884223 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2455b36a5af74869a1f30f567bc5866e, State: Initialized, Role: FOLLOWER
I20260810 15:02:11.884343 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } }
I20260810 15:02:11.884405 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:11.884434 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:11.884480 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:11.885069 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } }
I20260810 15:02:11.885174 477 leader_election.cc:302] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2455b36a5af74869a1f30f567bc5866e; no voters:
I20260810 15:02:11.885344 477 leader_election.cc:288] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:11.885376 490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:11.885488 490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [term 1 LEADER]: Becoming Leader. State: Replica: 2455b36a5af74869a1f30f567bc5866e, State: Running, Role: LEADER
I20260810 15:02:11.885571 490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } }
I20260810 15:02:11.885600 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:11.886031 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:11.881006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41589 (local address 127.28.193.254:35699)
0810 15:02:11.881189 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:11.881199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:11.881226 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:02:11.881657 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:11.881664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:11.881740 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:11.881884 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:11.882116 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:11.883027 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:11.883797 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:11.884241 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:11.884341 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:11.885198 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:11.885207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:11.885208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:11.885400 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:11.885441 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:11.885458 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:11.885540 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:11.885779 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 15:02:11.885810 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260810 15:02:11.886039 490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2455b36a5af74869a1f30f567bc5866e. Latest consensus state: current_term: 1 leader_uuid: "2455b36a5af74869a1f30f567bc5866e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } } }
I20260810 15:02:11.886224 490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:11.886049 493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2455b36a5af74869a1f30f567bc5866e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2455b36a5af74869a1f30f567bc5866e" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 35699 } } }
I20260810 15:02:11.886334 493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:11.887058 504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:11.887465 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:11.887533 504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:11.889082 504 catalog_manager.cc:1347] Generated new cluster ID: ea0c7d57a6d1426aa815c7d9ce3d6826
I20260810 15:02:11.889130 504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:11.892271 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:11.888949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49207 (local address 127.28.193.254:35699)
0810 15:02:11.888984 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:11.888986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:11.889072 (+ 86us) server_negotiation.cc:407] Connection header received
0810 15:02:11.889118 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:11.889119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:11.889128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:11.889187 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:11.889322 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:11.890053 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:11.891014 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:11.891317 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:11.891388 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:11.892040 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:11.892042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:11.892042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:11.892046 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:11.892061 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:11.892065 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:11.892125 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:11.892189 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 15:02:11.892200 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:11.920895 504 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:11.921489 504 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:11.924070 504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2455b36a5af74869a1f30f567bc5866e: Generated new TSK 0
I20260810 15:02:11.924237 504 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:11.931874 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 373
W20260810 15:02:11.943001 351 connection.cc:535] client connection to 127.28.193.254:35699 recv error: Network error: failed to read from TLS socket (remote: 127.28.193.254:35699): Connection reset by peer (error 104)
2026-08-10T15:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:11 dist-test-slave-cvwc krb5kdc[529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:11 dist-test-slave-cvwc krb5kdc[529](info): set up 2 sockets
Aug 10 15:02:11 dist-test-slave-cvwc krb5kdc[529](info): commencing operation
krb5kdc: starting...
W20260810 15:02:13.095759 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:13 dist-test-slave-cvwc krb5kdc[529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:13.212883 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36373
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:13.296871 553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:13.297096 553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:13.297119 553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:13.298288 553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:13.298328 553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:13.298341 553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:13.298353 553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:13.298370 553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:13.299858 553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:36373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:13.300067 553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:13.300252 553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:13.305095 563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:13.305122 553 server_base.cc:1034] running on GCE node
W20260810 15:02:13.305130 566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:13.305095 564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:13.305599 553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:13.305908 553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:13.307060 553 hybrid_clock.cc:648] HybridClock initialized: now 1786374133307042 us; error 30 us; skew 500 ppm
Aug 10 15:02:13 dist-test-slave-cvwc krb5kdc[529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374133, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:13.314172 553 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:13.314656 553 webserver.cc:460] Webserver started at http://127.28.193.254:44299/ using document root <none> and password file <none>
I20260810 15:02:13.314868 553 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:13.314909 553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:13.315007 553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:13.316087 553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "ba106040550542538f93903b1e806c24"
format_stamp: "Formatted at 2026-08-10 15:02:13 on dist-test-slave-cvwc"
server_key: "20b7b7991bfcb789c8f02bdedc76d35e"
server_key_iv: "8a3c3773118b20a367129db432e2cd5f"
server_key_version: "encryptionkey@0"
I20260810 15:02:13.316411 553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "ba106040550542538f93903b1e806c24"
format_stamp: "Formatted at 2026-08-10 15:02:13 on dist-test-slave-cvwc"
server_key: "20b7b7991bfcb789c8f02bdedc76d35e"
server_key_iv: "8a3c3773118b20a367129db432e2cd5f"
server_key_version: "encryptionkey@0"
I20260810 15:02:13.317760 553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:02:13.318570 574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:13.318723 553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:02:13.318816 553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "ba106040550542538f93903b1e806c24"
format_stamp: "Formatted at 2026-08-10 15:02:13 on dist-test-slave-cvwc"
server_key: "20b7b7991bfcb789c8f02bdedc76d35e"
server_key_iv: "8a3c3773118b20a367129db432e2cd5f"
server_key_version: "encryptionkey@0"
I20260810 15:02:13.318881 553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:13.322247 553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:13.324291 553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:13.324404 553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:13.328215 553 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:36373
I20260810 15:02:13.328260 660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:36373 every 8 connection(s)
I20260810 15:02:13.328579 553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:13.329468 661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:13.331816 661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24: Bootstrap starting.
I20260810 15:02:13.332396 661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:13.332660 661 log.cc:824] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:13.333364 661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24: No bootstrap required, opened a new log
I20260810 15:02:13.334976 661 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } }
I20260810 15:02:13.335120 661 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:13.335155 661 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba106040550542538f93903b1e806c24, State: Initialized, Role: FOLLOWER
I20260810 15:02:13.335292 661 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } }
I20260810 15:02:13.335376 661 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:13.335408 661 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:13.335456 661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:13.336287 661 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } }
I20260810 15:02:13.336385 661 leader_election.cc:302] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba106040550542538f93903b1e806c24; no voters:
I20260810 15:02:13.336546 661 leader_election.cc:288] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:13.336592 664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:13.336779 664 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [term 1 LEADER]: Becoming Leader. State: Replica: ba106040550542538f93903b1e806c24, State: Running, Role: LEADER
I20260810 15:02:13.336788 661 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:13.336951 664 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } }
I20260810 15:02:13.337313 666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba106040550542538f93903b1e806c24. Latest consensus state: current_term: 1 leader_uuid: "ba106040550542538f93903b1e806c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } } }
I20260810 15:02:13.337396 666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:13.337567 665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba106040550542538f93903b1e806c24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba106040550542538f93903b1e806c24" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36373 } } }
I20260810 15:02:13.337596 677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:13.337882 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 553
I20260810 15:02:13.337659 665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:13.338017 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:13.338196 29447 external_mini_cluster.cc:1428] Setting key 0a9d9db331d69da3e2da01f4f65cf974
I20260810 15:02:13.338353 677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:13.340482 677 catalog_manager.cc:1347] Generated new cluster ID: e5633bd22c1843fc8ef8dc75660ba320
I20260810 15:02:13.340534 677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:13.341030 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:13.338815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50861 (local address 127.28.193.254:36373)
0810 15:02:13.339180 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:13.339190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:13.339221 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:02:13.339661 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:13.339669 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:13.339696 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:13.339860 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:13.340512 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:13.340516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:13.340516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:13.340521 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:13.340569 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:13.340585 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:13.340785 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:13.340818 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 15:02:13.340859 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":196,"threads_started":1}
I20260810 15:02:13.341982 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:13.346745 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:13.343477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44027 (local address 127.28.193.254:36373)
0810 15:02:13.343509 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:13.343516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:13.345876 (+ 2360us) server_negotiation.cc:407] Connection header received
0810 15:02:13.345923 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:13.345924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:13.345929 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:13.345997 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:13.346546 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:13.346548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:13.346549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:13.346553 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:13.346568 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:13.346572 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:13.346630 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:13.346672 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:02:13.346684 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:02:13.364436 677 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:13.364600 677 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:13.375427 677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba106040550542538f93903b1e806c24: Generated new TSK 0
I20260810 15:02:13.375633 677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:13.382352 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 553
2026-08-10T15:02:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:13 dist-test-slave-cvwc krb5kdc[708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:13 dist-test-slave-cvwc krb5kdc[708](info): set up 2 sockets
Aug 10 15:02:13 dist-test-slave-cvwc krb5kdc[708](info): commencing operation
krb5kdc: starting...
W20260810 15:02:14.559736 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:14.682986 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33979
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:46873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:14.766004 732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:14.766211 732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:14.766242 732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:14.767378 732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:14.767421 732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:14.767439 732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:14.767455 732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:14.767469 732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:14.768965 732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:46873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:33979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:14.769220 732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:14.769429 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:14.774092 740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:14.774142 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:14.774075 741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:14.774225 732 server_base.cc:1034] running on GCE node
I20260810 15:02:14.774617 732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:14.774900 732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:14.776046 732 hybrid_clock.cc:648] HybridClock initialized: now 1786374134776033 us; error 29 us; skew 500 ppm
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374134, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:14.778541 732 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:14.778944 732 webserver.cc:460] Webserver started at http://127.28.193.254:43035/ using document root <none> and password file <none>
I20260810 15:02:14.779152 732 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:14.779196 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:14.779321 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:14.780378 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "bf9b359529844cb1bbc0d2c0235c91b6"
format_stamp: "Formatted at 2026-08-10 15:02:14 on dist-test-slave-cvwc"
server_key: "005a85ddb03df699609e2697507488b2"
server_key_iv: "204a9c060179c973189c24a1f235441c"
server_key_version: "encryptionkey@0"
I20260810 15:02:14.780706 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "bf9b359529844cb1bbc0d2c0235c91b6"
format_stamp: "Formatted at 2026-08-10 15:02:14 on dist-test-slave-cvwc"
server_key: "005a85ddb03df699609e2697507488b2"
server_key_iv: "204a9c060179c973189c24a1f235441c"
server_key_version: "encryptionkey@0"
I20260810 15:02:14.781991 732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:14.782725 757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:14.782868 732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:14.782954 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "bf9b359529844cb1bbc0d2c0235c91b6"
format_stamp: "Formatted at 2026-08-10 15:02:14 on dist-test-slave-cvwc"
server_key: "005a85ddb03df699609e2697507488b2"
server_key_iv: "204a9c060179c973189c24a1f235441c"
server_key_version: "encryptionkey@0"
I20260810 15:02:14.783016 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:14.791777 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:14.793380 732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:14.793504 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:14.797171 732 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:33979
I20260810 15:02:14.797215 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:33979 every 8 connection(s)
I20260810 15:02:14.797531 732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:14.798511 835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:14.798496 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 732
I20260810 15:02:14.798620 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:14.798815 29447 external_mini_cluster.cc:1428] Setting key 2a70aff79a17dcb34ab40cbd7a5ea298
I20260810 15:02:14.801543 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: Bootstrap starting.
I20260810 15:02:14.802266 835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:14.802559 835 log.cc:824] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:14.803253 835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: No bootstrap required, opened a new log
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:14.805565 835 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } }
I20260810 15:02:14.805759 835 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:14.805809 835 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9b359529844cb1bbc0d2c0235c91b6, State: Initialized, Role: FOLLOWER
I20260810 15:02:14.805958 835 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } }
I20260810 15:02:14.806061 835 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:14.806102 835 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:14.806144 835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:14.806739 835 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } }
I20260810 15:02:14.806855 835 leader_election.cc:302] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf9b359529844cb1bbc0d2c0235c91b6; no voters:
I20260810 15:02:14.807041 835 leader_election.cc:288] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:14.807083 842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:14.807238 842 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9b359529844cb1bbc0d2c0235c91b6, State: Running, Role: LEADER
I20260810 15:02:14.807303 835 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:14.807327 842 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } }
I20260810 15:02:14.807654 845 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } } }
I20260810 15:02:14.807861 845 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:14.807788 849 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf9b359529844cb1bbc0d2c0235c91b6. Latest consensus state: current_term: 1 leader_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf9b359529844cb1bbc0d2c0235c91b6" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 33979 } } }
I20260810 15:02:14.808090 849 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6 [sys.catalog]: This master's current role is: LEADER
W20260810 15:02:14.808961 875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:14.809002 875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:14.809057 860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:14.809183 860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:14.810139 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:14.799306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36109 (local address 127.28.193.254:33979)
0810 15:02:14.799462 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:14.799470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:14.799494 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:02:14.799946 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:14.799953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:14.799997 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:14.800104 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:14.800845 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:14.801759 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:14.802765 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:14.803243 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:14.803353 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:14.805872 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:14.805876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:14.805877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:14.805888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:14.808179 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:14.808586 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:14.808588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:14.808589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:14.808736 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:14.808946 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:14.808947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:14.808948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:14.809232 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:14.809403 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:14.809818 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 15:02:14.809954 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260810 15:02:14.810685 860 catalog_manager.cc:1347] Generated new cluster ID: 0ec925baa41b49f0b13eecedc9b9f0a2
I20260810 15:02:14.810737 860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:14.811074 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:14.823338 860 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:14.823830 860 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:14.824834 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:14.813981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45093 (local address 127.28.193.254:33979)
0810 15:02:14.814021 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:14.814023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:14.817867 (+ 3844us) server_negotiation.cc:407] Connection header received
0810 15:02:14.817917 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:14.817918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:14.817929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:14.817992 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:14.818594 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:14.819377 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:14.820292 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:14.820628 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:14.820695 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:14.822026 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:14.822028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:14.822029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:14.822036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:14.823708 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:14.824098 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:14.824099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:14.824099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:14.824160 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:14.824338 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:14.824339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:14.824339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:14.824460 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:14.824564 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:14.824656 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 15:02:14.824743 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:02:14.827504 860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf9b359529844cb1bbc0d2c0235c91b6: Generated new TSK 0
I20260810 15:02:14.827651 860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:14.830911 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 732
2026-08-10T15:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[889](info): set up 2 sockets
Aug 10 15:02:14 dist-test-slave-cvwc krb5kdc[889](info): commencing operation
krb5kdc: starting...
W20260810 15:02:16.034050 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:16 dist-test-slave-cvwc krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:16.150619 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37367
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:16.235133 906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:16.235494 906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:16.235549 906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:16.236771 906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:16.236820 906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:16.236833 906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:16.236845 906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:16.236858 906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:16.238422 906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:37367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:16.238700 906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:16.238929 906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:16.243813 914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:16.243896 913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:16.243978 906 server_base.cc:1034] running on GCE node
W20260810 15:02:16.243813 917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:16.244326 906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:16.244621 906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:16.245783 906 hybrid_clock.cc:648] HybridClock initialized: now 1786374136245765 us; error 29 us; skew 500 ppm
Aug 10 15:02:16 dist-test-slave-cvwc krb5kdc[889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374136, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:16.248294 906 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:16.248710 906 webserver.cc:460] Webserver started at http://127.28.193.254:33149/ using document root <none> and password file <none>
I20260810 15:02:16.248945 906 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:16.248998 906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:16.249122 906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:16.250226 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "4184075097454bdba8778dafb7eaf0da"
format_stamp: "Formatted at 2026-08-10 15:02:16 on dist-test-slave-cvwc"
server_key: "6608e9a62beee649c2969dfcd2daeded"
server_key_iv: "d263b75bd8f360b3ab03cfbba226cebd"
server_key_version: "encryptionkey@0"
I20260810 15:02:16.250581 906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "4184075097454bdba8778dafb7eaf0da"
format_stamp: "Formatted at 2026-08-10 15:02:16 on dist-test-slave-cvwc"
server_key: "6608e9a62beee649c2969dfcd2daeded"
server_key_iv: "d263b75bd8f360b3ab03cfbba226cebd"
server_key_version: "encryptionkey@0"
I20260810 15:02:16.251873 906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:02:16.252635 928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:16.252905 906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:02:16.253000 906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "4184075097454bdba8778dafb7eaf0da"
format_stamp: "Formatted at 2026-08-10 15:02:16 on dist-test-slave-cvwc"
server_key: "6608e9a62beee649c2969dfcd2daeded"
server_key_iv: "d263b75bd8f360b3ab03cfbba226cebd"
server_key_version: "encryptionkey@0"
I20260810 15:02:16.253116 906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:16.274524 906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:16.276713 906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:16.276831 906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:16.280737 906 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:37367
I20260810 15:02:16.280783 1013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:37367 every 8 connection(s)
I20260810 15:02:16.281121 906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:16.282035 1015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:16.284425 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da: Bootstrap starting.
I20260810 15:02:16.285006 1015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:16.285293 1015 log.cc:824] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:16.286006 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da: No bootstrap required, opened a new log
I20260810 15:02:16.286258 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 906
I20260810 15:02:16.286370 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:16.286531 29447 external_mini_cluster.cc:1428] Setting key 4c22c38c01c4cc63e8bcb7d6f8f0c7c7
I20260810 15:02:16.288085 1015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } }
I20260810 15:02:16.288264 1015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:16.288362 1015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4184075097454bdba8778dafb7eaf0da, State: Initialized, Role: FOLLOWER
I20260810 15:02:16.288522 1015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } }
I20260810 15:02:16.288595 1015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:16.288626 1015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:16.288674 1015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:16.289224 1015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } }
I20260810 15:02:16.289330 1015 leader_election.cc:302] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4184075097454bdba8778dafb7eaf0da; no voters:
I20260810 15:02:16.289489 1015 leader_election.cc:288] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:16.289532 1022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:16.289680 1022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [term 1 LEADER]: Becoming Leader. State: Replica: 4184075097454bdba8778dafb7eaf0da, State: Running, Role: LEADER
I20260810 15:02:16.289786 1015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:16.289847 1022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } }
I20260810 15:02:16.290220 1022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4184075097454bdba8778dafb7eaf0da. Latest consensus state: current_term: 1 leader_uuid: "4184075097454bdba8778dafb7eaf0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } } }
I20260810 15:02:16.290441 1022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:16.290374 1025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4184075097454bdba8778dafb7eaf0da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4184075097454bdba8778dafb7eaf0da" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 37367 } } }
I20260810 15:02:16.290827 1025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:16.291005 1043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:16.291589 1043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:16.293140 1043 catalog_manager.cc:1347] Generated new cluster ID: 5decafd6bdf842f7a9941b555e000be9
I20260810 15:02:16.293190 1043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:16.293489 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:16.287046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58941 (local address 127.28.193.254:37367)
0810 15:02:16.287278 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:16.287288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:16.287324 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:02:16.288051 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:16.288060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:16.288149 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:16.288303 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:16.288516 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:16.289905 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:16.290976 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:16.291501 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:16.291661 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:16.292724 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:16.292728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:16.292728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:16.292935 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:16.292984 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:16.292999 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:16.293101 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:16.293212 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 15:02:16.293280 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260810 15:02:16.294454 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:16.301008 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:16.295825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53305 (local address 127.28.193.254:37367)
0810 15:02:16.295870 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:16.295871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:16.297878 (+ 2007us) server_negotiation.cc:407] Connection header received
0810 15:02:16.297928 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:16.297929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:16.297940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:16.298001 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:16.298125 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:16.298853 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:16.299784 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:16.300074 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:16.300148 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:16.300776 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:16.300777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:16.300777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:16.300782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:16.300798 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:16.300810 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:16.300859 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:16.300908 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 15:02:16.300932 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 15:02:16.320089 1043 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:16.320628 1043 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:16.325042 1043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4184075097454bdba8778dafb7eaf0da: Generated new TSK 0
I20260810 15:02:16.325206 1043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:16.331918 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 906
2026-08-10T15:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1507 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:16 dist-test-slave-cvwc krb5kdc[1064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:16 dist-test-slave-cvwc krb5kdc[1064](info): set up 2 sockets
Aug 10 15:02:16 dist-test-slave-cvwc krb5kdc[1064](info): commencing operation
krb5kdc: starting...
W20260810 15:02:17.511189 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:17 dist-test-slave-cvwc krb5kdc[1064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:17.627997 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36791
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:36791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:17.710556 1083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:17.710758 1083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:17.710779 1083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:17.711854 1083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:17.711889 1083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:17.711901 1083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:17.711911 1083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:17.711927 1083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:17.713341 1083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:36791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:36791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:17.713531 1083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:17.713699 1083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:17.718309 1096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:17.718408 1083 server_base.cc:1034] running on GCE node
W20260810 15:02:17.718326 1101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:17.718294 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:17.718791 1083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:17.719087 1083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:17.720229 1083 hybrid_clock.cc:648] HybridClock initialized: now 1786374137720215 us; error 28 us; skew 500 ppm
Aug 10 15:02:17 dist-test-slave-cvwc krb5kdc[1064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374137, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:17.722599 1083 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:17.723017 1083 webserver.cc:460] Webserver started at http://127.28.193.254:34213/ using document root <none> and password file <none>
I20260810 15:02:17.723213 1083 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:17.723258 1083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:17.723387 1083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:17.724418 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "46dae1a305eb4a2bae2ad08020fc2523"
format_stamp: "Formatted at 2026-08-10 15:02:17 on dist-test-slave-cvwc"
server_key: "763fdd406b97ff28af85e91388443d11"
server_key_iv: "3e25b119ada2b05eeb8ef0625d09e358"
server_key_version: "encryptionkey@0"
I20260810 15:02:17.724761 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "46dae1a305eb4a2bae2ad08020fc2523"
format_stamp: "Formatted at 2026-08-10 15:02:17 on dist-test-slave-cvwc"
server_key: "763fdd406b97ff28af85e91388443d11"
server_key_iv: "3e25b119ada2b05eeb8ef0625d09e358"
server_key_version: "encryptionkey@0"
I20260810 15:02:17.726052 1083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:02:17.726712 1113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:17.726864 1083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:17.726953 1083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "46dae1a305eb4a2bae2ad08020fc2523"
format_stamp: "Formatted at 2026-08-10 15:02:17 on dist-test-slave-cvwc"
server_key: "763fdd406b97ff28af85e91388443d11"
server_key_iv: "3e25b119ada2b05eeb8ef0625d09e358"
server_key_version: "encryptionkey@0"
I20260810 15:02:17.727012 1083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:17.730219 1083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:17.732102 1083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:17.732229 1083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:17.735937 1083 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:36791
I20260810 15:02:17.735970 1190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:36791 every 8 connection(s)
I20260810 15:02:17.736270 1083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:17.737206 1194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:17.739488 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523: Bootstrap starting.
I20260810 15:02:17.740010 1194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:17.740249 1194 log.cc:824] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:17.740869 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523: No bootstrap required, opened a new log
I20260810 15:02:17.742457 1194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } }
I20260810 15:02:17.742594 1194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:02:17.742615 1194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46dae1a305eb4a2bae2ad08020fc2523, State: Initialized, Role: FOLLOWER
I20260810 15:02:17.742718 1194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } }
I20260810 15:02:17.742790 1194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:02:17.742817 1194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:02:17.742858 1194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:17.743031 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1083
I20260810 15:02:17.743144 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:17.743353 29447 external_mini_cluster.cc:1428] Setting key 5c15f76a41bdd50285afc339a26e173b
I20260810 15:02:17.743855 1194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } }
I20260810 15:02:17.743978 1194 leader_election.cc:302] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46dae1a305eb4a2bae2ad08020fc2523; no voters:
I20260810 15:02:17.744139 1194 leader_election.cc:288] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:02:17.744189 1201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:17.744393 1201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [term 1 LEADER]: Becoming Leader. State: Replica: 46dae1a305eb4a2bae2ad08020fc2523, State: Running, Role: LEADER
I20260810 15:02:17.744388 1194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:17.744547 1201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } }
I20260810 15:02:17.744885 1202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46dae1a305eb4a2bae2ad08020fc2523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } } }
I20260810 15:02:17.744954 1203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46dae1a305eb4a2bae2ad08020fc2523. Latest consensus state: current_term: 1 leader_uuid: "46dae1a305eb4a2bae2ad08020fc2523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46dae1a305eb4a2bae2ad08020fc2523" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 36791 } } }
I20260810 15:02:17.745038 1202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:17.745077 1203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:17.745689 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:17.743882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58933 (local address 127.28.193.254:36791)
0810 15:02:17.744061 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:17.744071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:17.744104 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:02:17.744232 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:17.744240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:17.744272 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:17.744423 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:17.745163 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:17.745167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:17.745167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:17.745172 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:17.745211 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:17.745225 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:17.745302 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:17.745448 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:02:17.745495 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260810 15:02:17.745997 1213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:17.746616 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:17.746766 1213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:17.748183 1213 catalog_manager.cc:1347] Generated new cluster ID: 67d280052a3d42a19930cb346eecdc39
I20260810 15:02:17.748238 1213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:17.758950 1200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:17.749991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36449 (local address 127.28.193.254:36791)
0810 15:02:17.750025 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:17.750027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:17.757955 (+ 7928us) server_negotiation.cc:407] Connection header received
0810 15:02:17.758020 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:17.758021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:17.758029 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:17.758086 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:17.758725 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:17.758727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:17.758727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:02:17.758731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:17.758748 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:02:17.758752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:17.758808 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:17.758861 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:02:17.758874 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:02:17.760102 1213 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:17.760258 1213 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:17.766618 1213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46dae1a305eb4a2bae2ad08020fc2523: Generated new TSK 0
I20260810 15:02:17.766800 1213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:02:17.771136 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1083
2026-08-10T15:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1435 ms)
[----------] 18 tests from AuthTokenIssuingTest (26163 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:17 dist-test-slave-cvwc krb5kdc[1237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:17 dist-test-slave-cvwc krb5kdc[1237](info): set up 2 sockets
Aug 10 15:02:17 dist-test-slave-cvwc krb5kdc[1237](info): commencing operation
krb5kdc: starting...
W20260810 15:02:18.931603 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:19.049638 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34241
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:19.131757 1255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:19.131985 1255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:19.132004 1255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:19.132022 1255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:19.133170 1255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:19.133212 1255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:19.133224 1255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:19.133235 1255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:19.133246 1255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:19.135008 1255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:19.135255 1255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:19.135478 1255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:19.140163 1267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:19.140153 1271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.140244 1255 server_base.cc:1034] running on GCE node
W20260810 15:02:19.140169 1268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.140648 1255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:19.140935 1255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:19.142078 1255 hybrid_clock.cc:648] HybridClock initialized: now 1786374139142061 us; error 29 us; skew 500 ppm
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:19.147593 1255 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:19.148062 1255 webserver.cc:460] Webserver started at http://127.28.193.254:46233/ using document root <none> and password file <none>
I20260810 15:02:19.148284 1255 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:19.148332 1255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:19.148460 1255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:19.149518 1255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "b813106b4def4d6c94661cf06de3eb65"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "e0e33b48461a367cead03083925af96e"
server_key_iv: "82788b6c9903f3a8c857baf06eaa5da9"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.149889 1255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "b813106b4def4d6c94661cf06de3eb65"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "e0e33b48461a367cead03083925af96e"
server_key_iv: "82788b6c9903f3a8c857baf06eaa5da9"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.151331 1255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:02:19.152093 1280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:19.152262 1255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:02:19.152339 1255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "b813106b4def4d6c94661cf06de3eb65"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "e0e33b48461a367cead03083925af96e"
server_key_iv: "82788b6c9903f3a8c857baf06eaa5da9"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.152405 1255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:19.163264 1255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:19.165274 1255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:19.165411 1255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:19.169344 1255 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34241
I20260810 15:02:19.169386 1372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34241 every 8 connection(s)
I20260810 15:02:19.169731 1255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:19.170614 1373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:19.172186 1373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.175264 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1255
I20260810 15:02:19.175374 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:19.175559 29447 external_mini_cluster.cc:1428] Setting key cac911626c301c56c0fa1aa9b870d344
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:19.182368 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.172877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34241 (local address 127.0.0.1:37530)
0810 15:02:19.173094 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.173144 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.173655 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.173937 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.173940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.174219 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.174381 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.174385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.175534 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.175535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.176261 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.176263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.176871 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.176872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.176931 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.177429 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.177437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.178943 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.180797 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.180804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.180806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.180989 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.181306 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.181307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.181307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.181400 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.181909 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.181915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.181981 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.182063 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.182198 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260810 15:02:19.182399 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.172958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.28.193.254:34241)
0810 15:02:19.173115 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.173125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.173238 (+ 113us) server_negotiation.cc:407] Connection header received
0810 15:02:19.173827 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.173829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.173849 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.173927 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.174472 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.175440 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.176344 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.176776 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.176879 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.179022 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.179031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.179032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.179039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.180719 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.181089 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.181093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.181093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.181208 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.181478 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.181478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.181479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.181835 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.181928 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.182117 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.182289 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260810 15:02:19.183163 1373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.183521 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.183404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33540)
0810 15:02:19.183451 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.183500 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 15:02:19.183651 1284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
I20260810 15:02:19.184085 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.183998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33552)
0810 15:02:19.184040 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.184064 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:02:19.184193 1373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:44719: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.28.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
I20260810 15:02:19.201604 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:44719
--webserver_interface=127.28.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:02:19.233430 1373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } attempt: 1
I20260810 15:02:19.234030 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.233888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33566)
0810 15:02:19.233954 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.233997 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:02:19.234380 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.234294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33576)
0810 15:02:19.234335 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.234360 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:02:19.234489 1373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:44719: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
W20260810 15:02:19.285066 1383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:19.285283 1383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:19.285300 1383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:19.285318 1383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:19.286490 1383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:19.286535 1383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:19.286546 1383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:19.286557 1383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:19.286568 1383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:19.287997 1383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:44719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:19.288187 1383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:19.288350 1383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:19.292984 1395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.293058 1383 server_base.cc:1034] running on GCE node
W20260810 15:02:19.292984 1398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:19.293031 1393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.293417 1383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:19.293691 1383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:19.294875 1383 hybrid_clock.cc:648] HybridClock initialized: now 1786374139294861 us; error 33 us; skew 500 ppm
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:19.297189 1383 init.cc:377] Logged in from keytab as kudu/127.28.193.253@KRBTEST.COM (short username kudu)
I20260810 15:02:19.297571 1383 webserver.cc:460] Webserver started at http://127.28.193.253:39521/ using document root <none> and password file <none>
I20260810 15:02:19.297784 1383 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:19.297829 1383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:19.297930 1383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:19.298938 1383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data/instance:
uuid: "260a5651a29c4914ba4ef14bffc50e6d"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "69c17bac0d8f67474f731beba6e18f2a"
server_key_iv: "328fc9e503135442dba6adf71587801b"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.299243 1383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal/instance:
uuid: "260a5651a29c4914ba4ef14bffc50e6d"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "69c17bac0d8f67474f731beba6e18f2a"
server_key_iv: "328fc9e503135442dba6adf71587801b"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.300462 1383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:02:19.301075 1405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:19.301218 1383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:19.301301 1383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "260a5651a29c4914ba4ef14bffc50e6d"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "69c17bac0d8f67474f731beba6e18f2a"
server_key_iv: "328fc9e503135442dba6adf71587801b"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.301365 1383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:19.302644 1373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } attempt: 2
I20260810 15:02:19.303015 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.302874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33578)
0810 15:02:19.302936 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.302978 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:02:19.303406 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.303299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33590)
0810 15:02:19.303349 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.303380 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 15:02:19.303495 1373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:44719: Network error: Client connection negotiation failed: client connection to 127.28.193.253:44719: connect: Connection refused (error 111)
I20260810 15:02:19.334256 1383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:19.336279 1383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:19.336402 1383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:19.340301 1383 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.253:44719
I20260810 15:02:19.340359 1488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.253:44719 every 8 connection(s)
I20260810 15:02:19.340682 1383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
I20260810 15:02:19.341554 1492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:19.343122 1492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.347360 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1383
I20260810 15:02:19.347457 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-1/wal/instance
I20260810 15:02:19.347646 29447 external_mini_cluster.cc:1428] Setting key 43eb518627a54d6d655931c18ccba500
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:19.351946 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.343870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.28.193.254:34241)
0810 15:02:19.343933 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.343935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.344176 (+ 241us) server_negotiation.cc:407] Connection header received
0810 15:02:19.344578 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.344579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.344592 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.344649 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.345237 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.345760 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.346703 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.346928 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.346999 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.349228 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.349229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.349229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.349236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.350678 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.351103 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.351103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.351104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.351153 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.351401 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.351402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.351402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.351496 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.351540 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.351817 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.351881 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:02:19.352224 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.343817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34241 (local address 127.0.0.1:37538)
0810 15:02:19.343992 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.344030 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.344488 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.344715 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.344719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.344979 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.345172 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.345177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.345829 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.345830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.346612 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.346615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.347006 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.347007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.347102 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.347607 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.347618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.349135 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.350757 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.350765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.350768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.351036 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.351204 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.351205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.351206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.351347 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.351545 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.351551 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.351701 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.351809 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.351996 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260810 15:02:19.352770 1492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.193.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.252@KRBTEST.COM" created.
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:19.361635 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.353039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33594)
0810 15:02:19.353085 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.353094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.353131 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.353349 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.353350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.353486 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.353527 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.353528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.354355 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.354356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.354834 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.354835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.355311 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.355312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.355371 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.355767 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.355771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.359156 (+ 3385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.360728 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.360729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.360730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.360883 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.361053 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.361053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.361054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.361104 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.361342 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.361343 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.361378 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.361439 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.361531 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:02:19.361703 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.353085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.28.193.253:44719)
0810 15:02:19.353230 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.353234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.353248 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:02:19.353263 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.353264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.353274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.353343 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.353587 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.354282 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.354905 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.355246 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.355307 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.359242 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.359252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.359252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.359260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.360654 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.360949 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.360952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.360952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.361001 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.361152 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.361153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.361153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.361285 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.361362 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.361468 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.361583 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260810 15:02:19.362255 1492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.362594 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.362491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40910)
0810 15:02:19.362528 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.362571 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:02:19.362731 1408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
I20260810 15:02:19.363139 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.363065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40924)
0810 15:02:19.363101 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.363124 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:02:19.363238 1492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:34791: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
I20260810 15:02:19.372956 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:34791
--webserver_interface=127.28.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:02:19.394646 1373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } attempt: 3
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:19.401532 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.395036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33610)
0810 15:02:19.395104 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.395114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.395163 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.395300 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.395301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.395467 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.395532 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.395534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.396170 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.396172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.396741 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.396744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.397300 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.397302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.397368 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.397876 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.397880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.399096 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.400680 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.400681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.400681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.400834 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.401030 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.401030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.401030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.401074 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.401313 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.401314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.401349 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.401394 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.401463 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 15:02:19.401595 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.395112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.28.193.253:44719)
0810 15:02:19.395172 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.395174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.395207 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:02:19.395235 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.395236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.395248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.395288 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.395610 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.396095 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.396830 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.397196 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.397286 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.399181 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.399182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.399182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.399191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.400603 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.400905 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.400906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.400906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.400968 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.401123 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.401124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.401124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.401258 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.401328 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.401424 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.401504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:02:19.402107 1373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.402418 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.402326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40938)
0810 15:02:19.402366 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.402394 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 15:02:19.402690 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.402625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40944)
0810 15:02:19.402656 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.402677 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:02:19.402755 1373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:34791: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
I20260810 15:02:19.412492 1492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } attempt: 1
I20260810 15:02:19.412922 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.412788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40952)
0810 15:02:19.412841 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.412894 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:02:19.413249 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.413171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40954)
0810 15:02:19.413208 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.413232 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:02:19.413328 1492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:34791: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
I20260810 15:02:19.433888 1373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } attempt: 1
I20260810 15:02:19.434402 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.434219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40958)
0810 15:02:19.434291 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.434344 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:02:19.434813 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.434718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40966)
0810 15:02:19.434761 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.434789 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:02:19.434947 1373 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:34791: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
W20260810 15:02:19.456938 1504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:19.457151 1504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:19.457168 1504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:19.457190 1504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:19.458372 1504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:19.458416 1504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:19.458428 1504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:19.458441 1504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:19.458451 1504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:19.459959 1504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34241,127.28.193.253:44719,127.28.193.252:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:19.460201 1504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:19.460418 1504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:19.465184 1510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.465220 1504 server_base.cc:1034] running on GCE node
W20260810 15:02:19.465217 1511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:19.465174 1513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:19.465641 1504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:19.465982 1504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:19.467137 1504 hybrid_clock.cc:648] HybridClock initialized: now 1786374139467117 us; error 32 us; skew 500 ppm
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:19.469506 1504 init.cc:377] Logged in from keytab as kudu/127.28.193.252@KRBTEST.COM (short username kudu)
I20260810 15:02:19.469973 1504 webserver.cc:460] Webserver started at http://127.28.193.252:45051/ using document root <none> and password file <none>
I20260810 15:02:19.470191 1504 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:19.470240 1504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:19.470368 1504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:19.471447 1504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data/instance:
uuid: "500bff982fdc4002a560f0c4a66dfa47"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "ab13caa6d606dd83b58621c6e69ec831"
server_key_iv: "59a4147f203e83928abfbc9467f29b8f"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.471788 1504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal/instance:
uuid: "500bff982fdc4002a560f0c4a66dfa47"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "ab13caa6d606dd83b58621c6e69ec831"
server_key_iv: "59a4147f203e83928abfbc9467f29b8f"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.473197 1504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:19.474090 1524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:19.474256 1504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:02:19.474346 1504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "500bff982fdc4002a560f0c4a66dfa47"
format_stamp: "Formatted at 2026-08-10 15:02:19 on dist-test-slave-cvwc"
server_key: "ab13caa6d606dd83b58621c6e69ec831"
server_key_iv: "59a4147f203e83928abfbc9467f29b8f"
server_key_version: "encryptionkey@0"
I20260810 15:02:19.474414 1504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:19.481495 1492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } attempt: 2
I20260810 15:02:19.481983 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.481834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40976)
0810 15:02:19.481904 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.481949 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:02:19.482262 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.482189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:40992)
0810 15:02:19.482224 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.482247 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:02:19.482343 1492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:34791: Network error: Client connection negotiation failed: client connection to 127.28.193.252:34791: connect: Connection refused (error 111)
I20260810 15:02:19.490026 1504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:19.491894 1504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:19.492012 1504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:19.496131 1504 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.252:34791
I20260810 15:02:19.496165 1600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.252:34791 every 8 connection(s)
I20260810 15:02:19.496515 1504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
I20260810 15:02:19.497419 1601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:19.498095 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1504
I20260810 15:02:19.498201 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786373869351853-29447-0/minicluster-data/master-2/wal/instance
I20260810 15:02:19.498397 29447 external_mini_cluster.cc:1428] Setting key 8139e08cfc2cf7a99fac0becccb4e21b
I20260810 15:02:19.499461 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:19.507767 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.500386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.28.193.254:34241)
0810 15:02:19.500444 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.500446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.500513 (+ 67us) server_negotiation.cc:407] Connection header received
0810 15:02:19.500568 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.500569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.500580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.500639 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.501247 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.501697 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.502550 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.502726 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.502775 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.505025 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.505027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.505027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.505033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.506478 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.506888 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.506890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.506890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.506947 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.507155 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.507156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.507156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.507270 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.507325 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.507595 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.507674 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:02:19.507947 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.500283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34241 (local address 127.0.0.1:37552)
0810 15:02:19.500417 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.500464 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.500521 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.500651 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.500654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.500999 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.501138 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.501145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.501795 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.501796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.502474 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.502475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.502793 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.502794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.502885 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.503426 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.503433 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.504946 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.506550 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.506557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.506559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.506806 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.507002 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.507003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.507003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.507099 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.507336 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.507342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.507475 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.507564 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.507719 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260810 15:02:19.508106 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.498999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.28.193.252:34791)
0810 15:02:19.499152 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.499166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.499190 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:02:19.499633 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.499640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.499688 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.499841 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.500603 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.501686 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.502427 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.502779 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.502885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.505501 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.505509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.505509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.505516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.506958 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.507270 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.507273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.507273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.507326 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.507498 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.507499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.507499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.507631 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.507786 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.507829 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.507899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260810 15:02:19.508612 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.510082 1373 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } attempt: 2
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:19.516306 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.508874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.28.193.254:34241)
0810 15:02:19.508971 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.508973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.508981 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:02:19.509001 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.509002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.509010 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.509065 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.509557 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.510208 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.511118 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.511311 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.511362 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.513577 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.513578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.513578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.513585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.515253 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515561 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.515562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.515563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.515617 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515805 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.515806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.515806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.515939 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.516120 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.516157 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.516221 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 15:02:19.516420 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.508931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.28.193.253:44719)
0810 15:02:19.508974 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.508976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.508983 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:02:19.509005 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.509006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.509014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.509065 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.509329 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.509967 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.510795 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.511118 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.511178 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.513521 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.513522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.513523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.513530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.515231 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515626 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.515627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.515627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.515683 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515878 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.515879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.515879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.516010 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.516073 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.516259 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.516336 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:19.516698 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.508825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:44719 (local address 127.0.0.1:33626)
0810 15:02:19.508866 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.508875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.508909 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.509067 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.509067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.509203 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.509259 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.509260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.510037 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.510038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.510712 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.510714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.511230 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.511231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.511296 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.511651 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.511655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.513424 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.515321 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.515322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.515521 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.515742 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.515747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.515808 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.516103 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.516103 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.516144 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.516530 (+ 386us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.516616 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:02:19.516824 1377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.510286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:41020)
0810 15:02:19.510318 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.510327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.510356 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.510526 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.510526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.510636 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.510664 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.510665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.511481 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.511482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.512210 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.512214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.512655 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.512655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.512712 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.513001 (+ 289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.513005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.514249 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.515829 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.515830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.515830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.516050 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.516225 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.516226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.516226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.516281 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.516549 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.516549 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.516588 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.516655 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.516732 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 15:02:19.516856 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.510378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.28.193.252:34791)
0810 15:02:19.510417 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.510419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.510426 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:02:19.510443 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.510443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.510452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.510510 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.510729 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.511388 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.512297 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.512579 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.512643 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.514322 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.514323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.514323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.514329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.515748 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.516115 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.516116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.516116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.516165 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.516342 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.516343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.516343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.516465 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.516521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.516691 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.516771 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:02:19.517482 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:19.519021 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: Bootstrap starting.
I20260810 15:02:19.519754 1373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:19.520071 1373 log.cc:824] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: Log is configured to *not* fsync() on all Append() calls
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:19.520939 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: No bootstrap required, opened a new log
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:19.523033 1373 raft_consensus.cc:348] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.523226 1373 raft_consensus.cc:729] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b813106b4def4d6c94661cf06de3eb65, State: Initialized, Role: FOLLOWER
I20260810 15:02:19.523360 1373 consensus_queue.cc:260] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.523660 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.517704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:41034)
0810 15:02:19.517767 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.517785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.517820 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.517922 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.517922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.518039 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.518062 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.518063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.518621 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.518622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.519077 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.519079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.519432 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.519433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.519478 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.519811 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.519816 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.520949 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.522673 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.522674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.522674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.522885 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.523060 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.523061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.523061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.523115 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.523419 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.523419 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.523457 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.523516 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.523602 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 15:02:19.523727 1373 sys_catalog.cc:564] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:19.523962 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.517774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.28.193.252:34791)
0810 15:02:19.517800 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.517802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.517809 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:02:19.517861 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.517861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.517869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.517918 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.518120 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.518558 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.519167 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.519375 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.519428 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.521034 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.521035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.521036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.521042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.522590 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.522949 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.522950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.522951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.523003 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.523165 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.523166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.523167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.523355 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.523427 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.523832 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.523897 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 15:02:19.524147 1609 sys_catalog.cc:455] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.524224 1609 sys_catalog.cc:458] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.524866 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.517221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.28.193.253:44719)
0810 15:02:19.517256 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.517257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.517265 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:02:19.517314 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.517315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.517324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.517369 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.518021 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.518669 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.519531 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.519800 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.519843 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.522175 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.522176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.522176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.522183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.523694 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.524079 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.524080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.524081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.524141 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.524367 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.524368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.524368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.524491 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.524658 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.524702 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.524770 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:19.525352 1601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: Bootstrap starting.
W20260810 15:02:19.525547 1627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:19.525593 1627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:19.526348 1601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:19.526644 1601 log.cc:824] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:19.527493 1601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: No bootstrap required, opened a new log
I20260810 15:02:19.526628 1255 master_runner.cc:186] Error getting master registration for 127.28.193.253:44719: OK, instance_id { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" instance_seqno: 1786374139339225 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:02:19.528831 1255 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:02:19.529608 1601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.529790 1601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500bff982fdc4002a560f0c4a66dfa47, State: Initialized, Role: FOLLOWER
I20260810 15:02:19.529913 1601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.530318 1601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:19.530381 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:19.531293 1330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b813106b4def4d6c94661cf06de3eb65"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37568
I20260810 15:02:19.531407 1330 raft_consensus.cc:482] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:02:19.531459 1330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:19.532083 1330 raft_consensus.cc:504] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.532291 1330 leader_election.cc:288] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [CANDIDATE]: Term 1 election: Requested vote from peers 260a5651a29c4914ba4ef14bffc50e6d (127.28.193.253:44719), 500bff982fdc4002a560f0c4a66dfa47 (127.28.193.252:34791)
I20260810 15:02:19.531121 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.532867 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.532655 1565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b813106b4def4d6c94661cf06de3eb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "500bff982fdc4002a560f0c4a66dfa47"
I20260810 15:02:19.533002 1565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:19.533947 1565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b813106b4def4d6c94661cf06de3eb65 in term 1.
I20260810 15:02:19.534209 1284 leader_election.cc:302] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500bff982fdc4002a560f0c4a66dfa47, b813106b4def4d6c94661cf06de3eb65; no voters:
I20260810 15:02:19.534338 1609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:19.534494 1609 raft_consensus.cc:686] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [term 1 LEADER]: Becoming Leader. State: Replica: b813106b4def4d6c94661cf06de3eb65, State: Running, Role: LEADER
I20260810 15:02:19.534579 1609 consensus_queue.cc:237] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
W20260810 15:02:19.534818 1657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:19.534862 1657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:19.536056 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b813106b4def4d6c94661cf06de3eb65. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.536221 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:19.536546 1504 master_runner.cc:186] Error getting master registration for 127.28.193.253:44719: OK, instance_id { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" instance_seqno: 1786374139339225 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:02:19.538358 1504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:02:19.541826 1662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:19.542047 1662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:19.550853 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.534148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.28.193.254:34241)
0810 15:02:19.534191 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.534205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.541883 (+ 7678us) server_negotiation.cc:407] Connection header received
0810 15:02:19.542003 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.542004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.542015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.542063 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.542759 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.543477 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.544624 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.544911 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.544971 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.547045 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.547047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.547047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.547055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.548963 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.550033 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.550034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.550034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.550082 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.550278 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.550279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.550279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.550412 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.550495 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.550690 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.550768 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:19.550982 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.536711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.28.193.253:44719)
0810 15:02:19.536749 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.536751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.542064 (+ 5313us) server_negotiation.cc:407] Connection header received
0810 15:02:19.542973 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.542974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.542985 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.543012 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.543710 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.544413 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.545463 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.545745 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.545800 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.547870 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.547871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.547872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.547878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.549383 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.549992 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.549993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.549994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.550065 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.550341 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.550342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.550342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.550469 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.550527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.550821 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.550902 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:19.552696 1565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [term 1 FOLLOWER]: Refusing update from remote peer b813106b4def4d6c94661cf06de3eb65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:02:19.553059 1658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:02:19.554661 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b813106b4def4d6c94661cf06de3eb65. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.554757 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.557580 1658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.557668 1658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:19.558039 1670 mvcc.cc:205] Tried to move back new op lower bound from 7316988475604992000 to 7316988475534336000. Current Snapshot: MvccSnapshot[applied={T|T < 7316988475604992000}]
I20260810 15:02:19.558202 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.558284 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.558401 1672 mvcc.cc:205] Tried to move back new op lower bound from 7316988475604992000 to 7316988475534336000. Current Snapshot: MvccSnapshot[applied={T|T < 7316988475604992000}]
I20260810 15:02:19.559190 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.536991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41048 (local address 127.28.193.252:34791)
0810 15:02:19.537030 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.537032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.542889 (+ 5857us) server_negotiation.cc:407] Connection header received
0810 15:02:19.542937 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.542938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.542949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.543008 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.551762 (+ 8754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.552380 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.553651 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.554087 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.554141 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.555770 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.555771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.555772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.555779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.557427 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.557895 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.557896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.557896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.557973 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.558246 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.558247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.558247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.558393 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.558501 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.559028 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.559108 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:02:19.559274 1662 catalog_manager.cc:1347] Generated new cluster ID: cc19f5422e1944ca80efff00b1823eab
I20260810 15:02:19.559320 1662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:19.570884 1662 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:19.571623 1662 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:19.573482 1492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } attempt: 3
I20260810 15:02:19.575726 1662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65: Generated new TSK 0
I20260810 15:02:19.575912 1662 catalog_manager.cc:1514] Initializing in-progress tserver states...
Aug 10 15:02:19 dist-test-slave-cvwc krb5kdc[1237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374139, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:19.586198 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.573782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:34791 (local address 127.0.0.1:41054)
0810 15:02:19.573829 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:19.573840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:19.573886 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:19.574023 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:19.574024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:19.574213 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:19.574262 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.574263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.575174 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.575175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.575868 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:19.575872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.576356 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.576358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:19.576429 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.577861 (+ 1432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:19.577866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:19.583207 (+ 5341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:19.585085 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.585087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.585087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.585307 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.585519 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:19.585519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:19.585520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:19.585587 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:19.585875 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:19.585876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:19.585926 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:02:19.585998 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:02:19.586097 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:02:19.586542 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.573820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.28.193.252:34791)
0810 15:02:19.573885 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.573887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.573934 (+ 47us) server_negotiation.cc:407] Connection header received
0810 15:02:19.573957 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.573958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.573968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.574019 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.574338 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.575102 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.575968 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.576279 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.576346 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.583300 (+ 6954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:19.583302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:19.583302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:19.583325 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:19.584993 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.585381 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.585382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.585382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.585457 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:19.585638 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:19.585639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:19.585640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:19.585811 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:19.585885 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.586392 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.586468 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:02:19.588050 1492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: Bootstrap starting.
I20260810 15:02:19.588805 1492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:19.589120 1492 log.cc:824] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:19.591053 1492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: No bootstrap required, opened a new log
I20260810 15:02:19.593079 1492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.593227 1492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 260a5651a29c4914ba4ef14bffc50e6d, State: Initialized, Role: FOLLOWER
I20260810 15:02:19.593309 1492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } }
I20260810 15:02:19.593650 1492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:19.593729 1446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b813106b4def4d6c94661cf06de3eb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "260a5651a29c4914ba4ef14bffc50e6d"
I20260810 15:02:19.593859 1446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:19.594672 1446 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b813106b4def4d6c94661cf06de3eb65 in term 1.
I20260810 15:02:19.594893 1444 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [term 1 FOLLOWER]: Refusing update from remote peer b813106b4def4d6c94661cf06de3eb65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:02:19.595376 1658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:02:19.601387 1378 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.600447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.28.193.254:34241)
0810 15:02:19.600489 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.600491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.600791 (+ 300us) server_negotiation.cc:407] Connection header received
0810 15:02:19.600841 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.600842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.600862 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.600921 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.601322 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37580: BlockingRecv error: recv got EOF from 127.0.0.1:37580 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:02:19.601989 1680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.602046 1698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b813106b4def4d6c94661cf06de3eb65. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.602092 1680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.602156 1680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:19.602195 1680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.602099 1698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:19.602654 1701 mvcc.cc:205] Tried to move back new op lower bound from 7316988475697184768 to 7316988475534336000. Current Snapshot: MvccSnapshot[applied={T|T < 7316988475604992000}]
I20260810 15:02:19.603811 1695 catalog_manager.cc:1259] Loaded cluster ID: cc19f5422e1944ca80efff00b1823eab
I20260810 15:02:19.603857 1695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: loading cluster ID for follower catalog manager: success
I20260810 15:02:19.605078 1695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: acquiring CA information for follower catalog manager: success
I20260810 15:02:19.605405 1695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 260a5651a29c4914ba4ef14bffc50e6d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 15:02:19.606365 1496 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.605684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.28.193.253:44719)
0810 15:02:19.605744 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.605746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.605925 (+ 179us) server_negotiation.cc:407] Connection header received
0810 15:02:19.605945 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.605946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.605955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.605996 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.606311 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:02:19.607219 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.606614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.28.193.252:34791)
0810 15:02:19.606652 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.606654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.606797 (+ 143us) server_negotiation.cc:407] Connection header received
0810 15:02:19.606818 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.606819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.606828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.606867 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.607163 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41056: BlockingRecv error: recv got EOF from 127.0.0.1:41056 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:02:19.611935 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.608730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.28.193.253:44719)
0810 15:02:19.608762 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.608763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.608935 (+ 172us) server_negotiation.cc:407] Connection header received
0810 15:02:19.609026 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.609026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.609054 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.609093 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.609257 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.610001 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.611100 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.611367 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.611416 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.611610 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.611721 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.611762 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.611855 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.611871 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 15:02:19.615229 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.614456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.28.193.252:34791)
0810 15:02:19.614493 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.614495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.614732 (+ 237us) server_negotiation.cc:407] Connection header received
0810 15:02:19.614831 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.614832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.614849 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.614876 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.615183 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41070: BlockingRecv error: recv got EOF from 127.0.0.1:41070 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:19.620563 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.617414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.28.193.253:44719)
0810 15:02:19.617446 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.617448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.617670 (+ 222us) server_negotiation.cc:407] Connection header received
0810 15:02:19.617808 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.617809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.617818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.617843 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.618016 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.618704 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.619767 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.620041 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.620085 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.620288 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.620321 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.620349 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.620476 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.620497 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:02:19.623606 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.622826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.28.193.252:34791)
0810 15:02:19.622861 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.622862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.623106 (+ 244us) server_negotiation.cc:407] Connection header received
0810 15:02:19.623194 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.623195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.623205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.623231 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.623560 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41076: BlockingRecv error: recv got EOF from 127.0.0.1:41076 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:02:19.629681 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.626654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.28.193.253:44719)
0810 15:02:19.626689 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.626690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.626916 (+ 226us) server_negotiation.cc:407] Connection header received
0810 15:02:19.627009 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.627010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.627020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.627046 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.627209 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.627899 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.628917 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.629178 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.629221 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.629429 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.629462 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.629491 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.629594 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.629612 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:02:19.632887 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.632082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.28.193.252:34791)
0810 15:02:19.632118 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.632120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.632348 (+ 228us) server_negotiation.cc:407] Connection header received
0810 15:02:19.632459 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.632460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.632471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.632499 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.632838 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41090: BlockingRecv error: recv got EOF from 127.0.0.1:41090 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:19.641251 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.638091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.28.193.253:44719)
0810 15:02:19.638128 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.638129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.638368 (+ 239us) server_negotiation.cc:407] Connection header received
0810 15:02:19.638502 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.638503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.638513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.638541 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.638710 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.639403 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.640474 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.640755 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.640799 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.640997 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.641031 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.641058 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.641161 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.641179 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:02:19.644351 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.643522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.28.193.252:34791)
0810 15:02:19.643560 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.643562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.643812 (+ 250us) server_negotiation.cc:407] Connection header received
0810 15:02:19.643909 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.643911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.643922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.643952 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.644300 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41094: BlockingRecv error: recv got EOF from 127.0.0.1:41094 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:02:19.656785 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.653576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.28.193.253:44719)
0810 15:02:19.653628 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.653629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.653916 (+ 287us) server_negotiation.cc:407] Connection header received
0810 15:02:19.654015 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.654016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.654027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.654054 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.654226 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.654937 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.655997 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.656284 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.656329 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.656534 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.656570 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.656596 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.656688 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.656707 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:02:19.659922 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.659093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.28.193.252:34791)
0810 15:02:19.659133 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.659135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.659388 (+ 253us) server_negotiation.cc:407] Connection header received
0810 15:02:19.659484 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.659486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.659499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.659527 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.659866 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41100: BlockingRecv error: recv got EOF from 127.0.0.1:41100 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:02:19.680944 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.677418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.28.193.253:44719)
0810 15:02:19.677466 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.677468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.677815 (+ 347us) server_negotiation.cc:407] Connection header received
0810 15:02:19.677933 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.677934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.677946 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.677978 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.678237 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.679001 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.680132 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.680434 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.680478 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.680681 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.680716 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.680741 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.680839 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.680860 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 15:02:19.684198 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.683333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.28.193.252:34791)
0810 15:02:19.683375 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.683376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.683555 (+ 179us) server_negotiation.cc:407] Connection header received
0810 15:02:19.683640 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.683641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.683652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.683679 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.684139 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41106: BlockingRecv error: recv got EOF from 127.0.0.1:41106 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:02:19.720081 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.717595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.28.193.253:44719)
0810 15:02:19.717647 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.717649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.717802 (+ 153us) server_negotiation.cc:407] Connection header received
0810 15:02:19.717896 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.717897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.717910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.717964 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.718102 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.718613 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.719415 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.719606 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.719669 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.719825 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.719854 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.719898 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.719966 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.719987 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 15:02:19.722486 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.721947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.28.193.252:34791)
0810 15:02:19.721989 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.721990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.721998 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:02:19.722024 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.722025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.722036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.722091 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.722439 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:02:19.790380 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.787849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.28.193.253:44719)
0810 15:02:19.787907 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.787909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.788019 (+ 110us) server_negotiation.cc:407] Connection header received
0810 15:02:19.788098 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.788099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.788111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.788171 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.788317 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.788818 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.789618 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.789838 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.789903 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.790045 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.790078 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.790126 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.790267 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.790295 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 15:02:19.792562 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.792024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.28.193.252:34791)
0810 15:02:19.792087 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.792089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.792137 (+ 48us) server_negotiation.cc:407] Connection header received
0810 15:02:19.792193 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.792194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.792203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.792247 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.792518 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41132: BlockingRecv error: recv got EOF from 127.0.0.1:41132 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:02:19.924126 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:19.921760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.28.193.253:44719)
0810 15:02:19.921820 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.921822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.921880 (+ 58us) server_negotiation.cc:407] Connection header received
0810 15:02:19.921960 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.921961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.921973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.922022 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:19.922163 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.922655 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.923422 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:19.923615 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:19.923726 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:19.923906 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:19.923939 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:19.923993 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:19.924045 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 15:02:19.924062 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 15:02:19.926534 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:19.925896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.28.193.252:34791)
0810 15:02:19.925959 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:19.925961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:19.925995 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:02:19.926079 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:19.926080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:19.926093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:19.926153 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:19.926476 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41134: BlockingRecv error: recv got EOF from 127.0.0.1:41134 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:02:20.150799 1875 sys_catalog.cc:455] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b813106b4def4d6c94661cf06de3eb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b813106b4def4d6c94661cf06de3eb65" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34241 } } peers { permanent_uuid: "260a5651a29c4914ba4ef14bffc50e6d" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 44719 } } peers { permanent_uuid: "500bff982fdc4002a560f0c4a66dfa47" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 34791 } } }
I20260810 15:02:20.150955 1875 sys_catalog.cc:458] T 00000000000000000000000000000000 P b813106b4def4d6c94661cf06de3eb65 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:20.186066 1496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.183816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.28.193.253:44719)
0810 15:02:20.183886 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.183889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.183902 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:02:20.183983 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.183984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.183996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.184041 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.184179 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.184662 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.185428 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.185613 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.185670 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.185837 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.185866 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.185908 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.185965 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.185985 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 15:02:20.188520 1605 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:20.187679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.28.193.252:34791)
0810 15:02:20.187747 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.187749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.187891 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:02:20.187962 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.187963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.187976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.188142 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:20.188469 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41150: BlockingRecv error: recv got EOF from 127.0.0.1:41150 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:02:20.535265 1657 catalog_manager.cc:1259] Loaded cluster ID: cc19f5422e1944ca80efff00b1823eab
I20260810 15:02:20.535349 1657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: loading cluster ID for follower catalog manager: success
I20260810 15:02:20.536170 1657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: acquiring CA information for follower catalog manager: success
I20260810 15:02:20.536482 1657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 500bff982fdc4002a560f0c4a66dfa47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:02:20.704146 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.701680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.28.193.253:44719)
0810 15:02:20.701819 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.701821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.701831 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:02:20.701942 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.701944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.701959 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.702038 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.702225 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.702732 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.703513 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.703715 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.703771 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.703928 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.703957 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.703989 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.704032 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.704049 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260810 15:02:20.708609 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.705780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.28.193.252:34791)
0810 15:02:20.705888 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.705890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.705899 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:02:20.705947 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.705948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.705998 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.706064 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.706147 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.706617 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.707695 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.707987 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.708068 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.708233 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.708386 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.708467 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.708491 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.708516 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260810 15:02:20.713105 1904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.710285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.28.193.253:44719)
0810 15:02:20.710333 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.710335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.710389 (+ 54us) server_negotiation.cc:407] Connection header received
0810 15:02:20.710457 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.710459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.710471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.710523 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.710654 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.711388 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712407 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.712687 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712741 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.712880 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.712913 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.712958 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.713005 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.713026 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:02:20.713312 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.710445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.28.193.252:34791)
0810 15:02:20.710479 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.710480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.710639 (+ 159us) server_negotiation.cc:407] Connection header received
0810 15:02:20.710702 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.710703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.710713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.710774 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.710931 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.711594 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712579 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.712853 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712890 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.713070 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.713101 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.713127 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.713227 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.713248 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:02:20.713587 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:20.710581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.28.193.254:34241)
0810 15:02:20.710793 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:20.710795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:20.710803 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:02:20.710888 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:20.710889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:20.710939 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:20.711007 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:20.711132 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.711621 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712353 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:20.712654 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:20.712708 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:20.712942 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:20.713051 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:20.713101 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:20.713500 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 15:02:20.713528 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":57,"threads_started":1}
I20260810 15:02:20.715276 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1255
I20260810 15:02:20.726837 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1383
I20260810 15:02:20.740079 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1504
2026-08-10T15:02:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2970 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:02:20 dist-test-slave-cvwc krb5kdc[1937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:02:20 dist-test-slave-cvwc krb5kdc[1937](info): set up 2 sockets
Aug 10 15:02:20 dist-test-slave-cvwc krb5kdc[1937](info): commencing operation
krb5kdc: starting...
W20260810 15:02:21.920956 29447 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/test-user.keytab.
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.193.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
Entry for principal kudu/127.28.193.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab.
I20260810 15:02:22.038421 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34031
--webserver_interface=127.28.193.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:02:22.120134 1954 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:22.120352 1954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:22.120378 1954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:22.120402 1954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:22.121547 1954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:22.121587 1954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:22.121606 1954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:22.121622 1954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:22.121637 1954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:22.123119 1954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.254.keytab
--principal=kudu/127.28.193.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.254:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.193.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:22.123369 1954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:22.123588 1954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:22.128337 1961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:22.128327 1963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:22.128327 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:22.128553 1954 server_base.cc:1034] running on GCE node
I20260810 15:02:22.128808 1954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:22.129117 1954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:22.130261 1954 hybrid_clock.cc:648] HybridClock initialized: now 1786374142130243 us; error 42 us; skew 500 ppm
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:22.132746 1954 init.cc:377] Logged in from keytab as kudu/127.28.193.254@KRBTEST.COM (short username kudu)
I20260810 15:02:22.133206 1954 webserver.cc:460] Webserver started at http://127.28.193.254:41111/ using document root <none> and password file <none>
I20260810 15:02:22.133420 1954 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:22.133468 1954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:22.133597 1954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:22.134753 1954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data/instance:
uuid: "a4c1e6fa9ec04542a00e021b5f7c8837"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "58dfe17da71c335d3cabee1d7c20531d"
server_key_iv: "f71dfafcf7b188915366ed6d327d7e59"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.135105 1954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance:
uuid: "a4c1e6fa9ec04542a00e021b5f7c8837"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "58dfe17da71c335d3cabee1d7c20531d"
server_key_iv: "f71dfafcf7b188915366ed6d327d7e59"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.136674 1954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:02:22.137519 1979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:22.137735 1954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:02:22.137837 1954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal
uuid: "a4c1e6fa9ec04542a00e021b5f7c8837"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "58dfe17da71c335d3cabee1d7c20531d"
server_key_iv: "f71dfafcf7b188915366ed6d327d7e59"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.137912 1954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:22.175173 1954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:22.177655 1954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:22.177834 1954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:22.181993 1954 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.254:34031
I20260810 15:02:22.182024 2061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.254:34031 every 8 connection(s)
I20260810 15:02:22.182413 1954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/data/info.pb
I20260810 15:02:22.183328 2062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:22.183629 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 1954
I20260810 15:02:22.183724 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-0/wal/instance
I20260810 15:02:22.183955 29447 external_mini_cluster.cc:1428] Setting key 72f5cb578d3619771681c437560a7937
I20260810 15:02:22.185389 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.253@KRBTEST.COM" created.
I20260810 15:02:22.195150 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.186365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.28.193.254:34031)
0810 15:02:22.186524 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.186530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.186573 (+ 43us) server_negotiation.cc:407] Connection header received
0810 15:02:22.186996 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.186999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.187021 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.187115 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.187632 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.188914 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.189779 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.190252 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.190369 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.192266 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.192274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.192274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.192282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.193654 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.194009 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.194012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.194012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.194117 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.194294 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.194294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.194294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.194528 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.194605 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.194770 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.194897 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260810 15:02:22.195153 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.186185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34031 (local address 127.0.0.1:57020)
0810 15:02:22.186336 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.186378 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.186846 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.187129 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.187132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.187368 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.187540 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.187544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.189007 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.189008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.189673 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.189674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.190330 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.190331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.190384 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.190856 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.190865 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.192187 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.193742 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.193751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.193753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.193942 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.194165 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.194165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.194166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.194246 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.194581 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.194585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.194641 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.194723 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.194950 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":100,"thread_start_us":65,"threads_started":1}
I20260810 15:02:22.195796 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:22.196089 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.195993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53674)
0810 15:02:22.196031 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.196071 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:02:22.196213 1980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
I20260810 15:02:22.196594 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.196514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53688)
0810 15:02:22.196548 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.196572 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260810 15:02:22.196722 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:46001: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
Entry for principal kudu/127.28.193.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab.
I20260810 15:02:22.209116 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:46001
--webserver_interface=127.28.193.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:02:22.245913 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } attempt: 1
I20260810 15:02:22.246444 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.246300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53696)
0810 15:02:22.246362 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.246412 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:02:22.246771 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.246688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53700)
0810 15:02:22.246723 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.246753 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:02:22.246882 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:46001: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
W20260810 15:02:22.291025 2069 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:22.291222 2069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:22.291239 2069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:22.291256 2069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:22.292349 2069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:22.292383 2069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:22.292395 2069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:22.292405 2069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:22.292416 2069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:22.293957 2069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.253.keytab
--principal=kudu/127.28.193.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.253:46001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.193.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:22.294167 2069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:22.294346 2069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:02:22.298919 2076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:22.298992 2077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:22.298947 2079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:22.299240 2069 server_base.cc:1034] running on GCE node
I20260810 15:02:22.299484 2069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:22.299772 2069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:22.300945 2069 hybrid_clock.cc:648] HybridClock initialized: now 1786374142300931 us; error 29 us; skew 500 ppm
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:22.303577 2069 init.cc:377] Logged in from keytab as kudu/127.28.193.253@KRBTEST.COM (short username kudu)
I20260810 15:02:22.304031 2069 webserver.cc:460] Webserver started at http://127.28.193.253:40021/ using document root <none> and password file <none>
I20260810 15:02:22.304250 2069 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:22.304302 2069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:22.304445 2069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:22.305492 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data/instance:
uuid: "8d00045f915546a49680be435a64bd75"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "a5b0769db9a7876dfac70752ceb64f3b"
server_key_iv: "b3ec6d3c3205819289755cc454edcbf7"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.305883 2069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal/instance:
uuid: "8d00045f915546a49680be435a64bd75"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "a5b0769db9a7876dfac70752ceb64f3b"
server_key_iv: "b3ec6d3c3205819289755cc454edcbf7"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.307242 2069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:02:22.308022 2087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:22.308208 2069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:02:22.308292 2069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal
uuid: "8d00045f915546a49680be435a64bd75"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "a5b0769db9a7876dfac70752ceb64f3b"
server_key_iv: "b3ec6d3c3205819289755cc454edcbf7"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.308354 2069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:22.315074 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } attempt: 2
I20260810 15:02:22.315487 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.315350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53710)
0810 15:02:22.315408 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.315451 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:02:22.315784 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.315719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53724)
0810 15:02:22.315748 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.315770 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 15:02:22.315879 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.253:46001: Network error: Client connection negotiation failed: client connection to 127.28.193.253:46001: connect: Connection refused (error 111)
I20260810 15:02:22.320654 2069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:22.322713 2069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:22.322845 2069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:22.326565 2069 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.253:46001
I20260810 15:02:22.326609 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.253:46001 every 8 connection(s)
I20260810 15:02:22.326923 2069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/data/info.pb
I20260810 15:02:22.327770 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:22.329291 2165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:22.334102 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2069
I20260810 15:02:22.334206 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-1/wal/instance
I20260810 15:02:22.334422 29447 external_mini_cluster.cc:1428] Setting key 8f9a5cb7938dad47d0ed2d78e49c6511
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:22.337942 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.329972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.28.193.254:34031)
0810 15:02:22.330033 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.330036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.330216 (+ 180us) server_negotiation.cc:407] Connection header received
0810 15:02:22.330713 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.330714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.330727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.330780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.331382 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.331873 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.332797 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.333008 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.333064 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.335283 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.335285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.335285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.335292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.336630 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.337019 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.337020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.337020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.337086 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.337329 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.337329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.337330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.337456 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.337520 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.337763 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.337856 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:02:22.338025 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.329934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34031 (local address 127.0.0.1:57022)
0810 15:02:22.330102 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.330140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.330615 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.330850 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.330854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.331102 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.331286 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.331290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.331951 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.331952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.332717 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.332720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.333076 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.333077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.333198 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.333610 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.333619 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.335181 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.336718 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.336725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.336727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.336943 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.337166 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.337166 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.337167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.337268 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.337513 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.337516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.337621 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.337695 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.337858 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260810 15:02:22.338550 2165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.193.252@KRBTEST.COM; defaulting to no policy
I20260810 15:02:22.344897 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.338751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53734)
0810 15:02:22.338956 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.338966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.339018 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.339147 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.339148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.339272 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.339314 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.339315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.340124 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.340125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.340652 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.340654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.341127 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.341128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.341209 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.341522 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.341527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.342762 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.344110 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.344110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.344111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.344259 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.344408 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.344408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.344409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.344449 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.344691 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.344691 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.344722 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.344762 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.344837 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 15:02:22.345000 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.338815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.28.193.253:46001)
0810 15:02:22.338968 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.338972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.339016 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:02:22.339070 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.339073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.339085 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.339140 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.339385 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.340053 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.340717 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.341020 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.341083 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.342856 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.342863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.342863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.342870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.344033 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.344313 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.344314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.344315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.344357 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.344493 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.344494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.344494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.344642 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.344705 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.344792 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.344886 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.193.252@KRBTEST.COM" created.
I20260810 15:02:22.345577 2165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:22.345921 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.345796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54614)
0810 15:02:22.345845 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.345896 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:02:22.346052 2097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
I20260810 15:02:22.346411 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.346323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54616)
0810 15:02:22.346361 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.346391 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 15:02:22.346508 2165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:33861: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
Entry for principal kudu/127.28.193.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab.
I20260810 15:02:22.358850 29447 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskILwtmC/build/release/bin/kudu
/tmp/dist-test-taskILwtmC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:33861
--webserver_interface=127.28.193.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.193.212:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:02:22.395736 2165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } attempt: 1
I20260810 15:02:22.396275 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.396117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54622)
0810 15:02:22.396181 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.396230 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:02:22.396672 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.396586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54632)
0810 15:02:22.396625 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.396652 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:02:22.396763 2165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:33861: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
I20260810 15:02:22.407038 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } attempt: 3
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:22.414108 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.407426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53738)
0810 15:02:22.407491 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.407503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.407569 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.407721 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.407721 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.407881 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.407949 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.407950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.408855 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.408856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.409375 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.409376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.409867 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.409867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.409924 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.410354 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.410359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.411568 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.413215 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.413216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.413217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.413382 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.413566 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.413566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.413567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.413614 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.413871 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.413872 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.413911 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.413962 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.414038 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:02:22.414275 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.407503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.28.193.253:46001)
0810 15:02:22.407568 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.407570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.407604 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:02:22.407634 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.407635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.407646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.407692 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.408012 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.408781 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.409440 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.409801 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.409872 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.411650 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.411652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.411652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.411660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.413138 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.413447 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.413448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.413449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.413509 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.413661 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.413662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.413662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.413816 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.413882 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.413996 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.414073 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:02:22.414597 2062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:02:22.414911 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.414815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54646)
0810 15:02:22.414860 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.414887 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:02:22.415217 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.415149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54660)
0810 15:02:22.415183 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.415204 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 15:02:22.415316 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:33861: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
W20260810 15:02:22.442373 2176 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:02:22.442575 2176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:02:22.442591 2176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:02:22.442610 2176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:02:22.443689 2176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:02:22.443723 2176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:02:22.443737 2176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:02:22.443747 2176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:02:22.443765 2176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:02:22.445175 2176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.193.212:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.193.254:34031,127.28.193.253:46001,127.28.193.252:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/krb5kdc/kudu_127.28.193.252.keytab
--principal=kudu/127.28.193.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.193.252:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.193.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 13:36:56 UTC on ccycloud.quasar-zxzunu.root.comops.site
build id 4577778
I20260810 15:02:22.445374 2176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:02:22.445541 2176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:02:22.446460 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } attempt: 1
I20260810 15:02:22.446925 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.446798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54662)
0810 15:02:22.446856 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.446895 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:02:22.447264 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.447167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54678)
0810 15:02:22.447207 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.447231 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 15:02:22.447348 2062 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:33861: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
W20260810 15:02:22.450242 2183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:22.450330 2176 server_base.cc:1034] running on GCE node
W20260810 15:02:22.450240 2184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:02:22.450237 2186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:02:22.450695 2176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:02:22.450994 2176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:02:22.452136 2176 hybrid_clock.cc:648] HybridClock initialized: now 1786374142452117 us; error 32 us; skew 500 ppm
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:02:22.454473 2176 init.cc:377] Logged in from keytab as kudu/127.28.193.252@KRBTEST.COM (short username kudu)
I20260810 15:02:22.454901 2176 webserver.cc:460] Webserver started at http://127.28.193.252:43945/ using document root <none> and password file <none>
I20260810 15:02:22.455109 2176 fs_manager.cc:359] Metadata directory not provided
I20260810 15:02:22.455156 2176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:02:22.455286 2176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:02:22.456324 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data/instance:
uuid: "ade09fb5788449ff814ed955f296c902"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "97aa0772761a9c8fdd75f3fffaa7d082"
server_key_iv: "8655c74a7344c5196277334e88f27836"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.456645 2176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal/instance:
uuid: "ade09fb5788449ff814ed955f296c902"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "97aa0772761a9c8fdd75f3fffaa7d082"
server_key_iv: "8655c74a7344c5196277334e88f27836"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.457947 2176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:02:22.458621 2199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:02:22.458756 2176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:02:22.458860 2176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data,/tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal
uuid: "ade09fb5788449ff814ed955f296c902"
format_stamp: "Formatted at 2026-08-10 15:02:22 on dist-test-slave-cvwc"
server_key: "97aa0772761a9c8fdd75f3fffaa7d082"
server_key_iv: "8655c74a7344c5196277334e88f27836"
server_key_version: "encryptionkey@0"
I20260810 15:02:22.458920 2176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:02:22.464915 2165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } attempt: 2
I20260810 15:02:22.465363 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.465219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54684)
0810 15:02:22.465283 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.465327 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:02:22.465664 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.465602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54692)
0810 15:02:22.465631 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.465650 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260810 15:02:22.465772 2165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.193.252:33861: Network error: Client connection negotiation failed: client connection to 127.28.193.252:33861: connect: Connection refused (error 111)
I20260810 15:02:22.478004 2176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:02:22.479866 2176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:02:22.479985 2176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:02:22.483906 2176 rpc_server.cc:307] RPC server started. Bound to: 127.28.193.252:33861
I20260810 15:02:22.483971 2278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.193.252:33861 every 8 connection(s)
I20260810 15:02:22.484277 2176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/data/info.pb
I20260810 15:02:22.485157 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:02:22.486801 2280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:22.494058 29447 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskILwtmC/build/release/bin/kudu as pid 2176
I20260810 15:02:22.494172 29447 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskILwtmC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786373869351853-29447-0/minicluster-data/master-2/wal/instance
I20260810 15:02:22.494378 29447 external_mini_cluster.cc:1428] Setting key bd802d585c30b6a5f75fd9d5d08dfaa8
I20260810 15:02:22.495627 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.487529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.28.193.254:34031)
0810 15:02:22.487619 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.487622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.487803 (+ 181us) server_negotiation.cc:407] Connection header received
0810 15:02:22.488304 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.488306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.488319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.488370 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.488984 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.489447 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.490420 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.490605 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.490664 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.492668 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.492669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.492670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.492677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.494017 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.494444 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.494446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.494446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.494513 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.494817 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.494819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.494819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.494979 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.495078 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.495440 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.495534 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:02:22.495818 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.487476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.254:34031 (local address 127.0.0.1:57036)
0810 15:02:22.487683 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.487722 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.488204 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.488449 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.488453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.488699 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.488895 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.488899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.489529 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.489529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.490338 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.490340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.490674 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.490675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.490797 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.491256 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.491265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.492579 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.494092 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.494100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.494103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.494374 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.494571 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.494571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.494572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.494683 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.495071 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.495078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.495250 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.495386 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.495584 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260810 15:02:22.496459 2280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:22.503405 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.496733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.253:46001 (local address 127.0.0.1:53750)
0810 15:02:22.496850 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.496859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.496902 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.497014 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.497015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.497163 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.497214 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.497215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.497829 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.497830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.498411 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.498413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.498816 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.498817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.498887 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.499220 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.499224 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.500739 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.502179 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.502181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.502386 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.502590 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.502591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.502672 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.503086 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.503087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.503129 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.503198 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.503305 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":31}
I20260810 15:02:22.503418 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.496792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.28.193.253:46001)
0810 15:02:22.496856 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.496858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.496890 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:02:22.496949 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.496950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.496959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.497003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.497274 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.497755 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.498497 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.498739 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.498802 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.500838 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.500840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.500840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.500848 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.502115 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502463 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.502464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.502464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.502526 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502845 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.502847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.502847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.503009 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.503086 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.503247 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.503322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:02:22.503408 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.494749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.28.193.252:33861)
0810 15:02:22.494933 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.494938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.494965 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:02:22.494987 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.494989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.495003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.495077 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.495793 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.496633 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.497578 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.498007 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.498081 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.500514 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.500522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.500523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.500531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.502051 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502474 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.502477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.502478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.502533 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.502736 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.502737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.502737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.502921 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.503063 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.503166 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.503258 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260810 15:02:22.503782 2280 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.252@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.254@KRBTEST.COM
I20260810 15:02:22.509946 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.504044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54724)
0810 15:02:22.504115 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.504124 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.504167 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.504308 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.504309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.504432 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.504459 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.504460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.505027 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.505028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.505480 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.505481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.505840 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.505840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.505884 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.506226 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.506231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.507573 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.509057 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.509058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.509059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.509210 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.509368 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.509369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.509369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.509420 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.509616 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.509617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.509653 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.509730 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.509828 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 15:02:22.509989 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.504077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.28.193.252:33861)
0810 15:02:22.504116 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.504118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.504156 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:02:22.504227 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.504228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.504257 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.504305 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.504516 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.504964 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.505543 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.505785 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.505842 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.507649 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.507650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.507650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.507656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.508991 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.509268 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.509268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.509269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.509317 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.509468 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.509468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.509469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.509566 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.509618 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.509758 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.509876 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":20}
I20260810 15:02:22.511044 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.504158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.28.193.254:34031)
0810 15:02:22.504198 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.504200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.504252 (+ 52us) server_negotiation.cc:407] Connection header received
0810 15:02:22.504330 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.504331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.504341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.504399 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.504878 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.505341 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.506042 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.506255 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.506317 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.508518 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.508520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.508521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.508529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.509909 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.510233 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.510234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.510234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.510290 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.510466 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.510467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.510467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.510594 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.510835 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.510881 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.510958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:22.511831 2280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: Bootstrap starting.
I20260810 15:02:22.512893 2280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:22.513224 2280 log.cc:824] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:22.514104 2280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: No bootstrap required, opened a new log
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.193.253@KRBTEST.COM
I20260810 15:02:22.516113 2280 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.516245 2280 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade09fb5788449ff814ed955f296c902, State: Initialized, Role: FOLLOWER
I20260810 15:02:22.516342 2280 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.516666 2293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.516732 2280 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:22.516789 2293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:02:22.518642 2307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:22.519011 2307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:22.519559 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.511704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.28.193.253:46001)
0810 15:02:22.511771 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.511773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.511785 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:02:22.511843 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.511844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.511877 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.511940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.512606 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.513105 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.513932 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.514128 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.514193 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.516709 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.516711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.516711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.516720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.518200 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.518620 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.518621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.518622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.518686 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.518880 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.518881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.518881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.519016 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.519178 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.519382 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.519468 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:02:22.520911 29447 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:02:22.519598 2176 master_runner.cc:186] Error getting master registration for 127.28.193.254:34031: OK, instance_id { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" instance_seqno: 1786374142180808 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:02:22.521859 2176 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:02:22.522496 2062 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } attempt: 2
I20260810 15:02:22.522028 2017 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57050
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.254@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:22.529544 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.522740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54730)
0810 15:02:22.522781 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.522792 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.522835 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.522978 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.522979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.523125 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.523165 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.523167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.523825 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.523826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.524321 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.524322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.524638 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.524639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.524690 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.525156 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.525161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.526717 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.528444 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.528445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.528445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.528675 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.528868 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.528869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.528869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.528936 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.529221 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.529222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.529271 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.529343 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.529440 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:02:22.529544 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.522834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.28.193.252:33861)
0810 15:02:22.522884 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.522886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.522893 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:02:22.522911 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.522912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.522921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.522968 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.523230 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.523733 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.524390 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.524583 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.524635 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.526809 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.526810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.526811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.526818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.528355 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.528745 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.528746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.528746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.528808 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.528991 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.528991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.528992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.529160 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.529221 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.529369 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.529451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:02:22.531272 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: Bootstrap starting.
I20260810 15:02:22.531910 2062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:22.532199 2062 log.cc:824] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:22.532932 2062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: No bootstrap required, opened a new log
I20260810 15:02:22.534017 2062 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.534137 2062 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c1e6fa9ec04542a00e021b5f7c8837, State: Initialized, Role: FOLLOWER
I20260810 15:02:22.534212 2062 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.534482 2311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.534546 2062 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:22.534603 2311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:22.534646 2017 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:02:22.534747 2017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:22.535532 2017 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
W20260810 15:02:22.536238 2330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:22.536345 2330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:22.537153 2242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ade09fb5788449ff814ed955f296c902"
I20260810 15:02:22.537282 2242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:22.538214 2242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c1e6fa9ec04542a00e021b5f7c8837 in term 1.
I20260810 15:02:22.538501 1954 master_runner.cc:186] Error getting master registration for 127.28.193.253:46001: OK, instance_id { permanent_uuid: "8d00045f915546a49680be435a64bd75" instance_seqno: 1786374142325476 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:02:22.540952 1954 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:02:22.538497 1984 leader_election.cc:302] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4c1e6fa9ec04542a00e021b5f7c8837, ade09fb5788449ff814ed955f296c902; no voters:
I20260810 15:02:22.541147 2017 leader_election.cc:288] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [CANDIDATE]: Term 1 election: Requested vote from peers 8d00045f915546a49680be435a64bd75 (127.28.193.253:46001), ade09fb5788449ff814ed955f296c902 (127.28.193.252:33861)
I20260810 15:02:22.541149 2311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:02:22.541385 2311 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [term 1 LEADER]: Becoming Leader. State: Replica: a4c1e6fa9ec04542a00e021b5f7c8837, State: Running, Role: LEADER
I20260810 15:02:22.541477 2311 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.542133 2311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c1e6fa9ec04542a00e021b5f7c8837. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.542230 2311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:22.542445 2339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:02:22.542655 2339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:02:22.544905 2242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [term 1 FOLLOWER]: Refusing update from remote peer a4c1e6fa9ec04542a00e021b5f7c8837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:02:22.545249 2332 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:02:22.546604 2293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c1e6fa9ec04542a00e021b5f7c8837. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.546694 2293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:22.551859 2340 mvcc.cc:205] Tried to move back new op lower bound from 7316988487861055488 to 7316988487851446272. Current Snapshot: MvccSnapshot[applied={T|T < 7316988487861055488}]
I20260810 15:02:22.552119 2339 catalog_manager.cc:1347] Generated new cluster ID: c486cef9643d477dbbfcbf37e194f999
I20260810 15:02:22.552172 2339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:02:22.552181 2332 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.552294 2332 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:22.552737 2293 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.552822 2293 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:22.553563 2343 mvcc.cc:205] Tried to move back new op lower bound from 7316988487861055488 to 7316988487851446272. Current Snapshot: MvccSnapshot[applied={T|T < 7316988487861055488}]
I20260810 15:02:22.556910 2165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } attempt: 3
I20260810 15:02:22.563429 2339 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:02:22.564127 2339 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:02:22.566643 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.550107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.28.193.254:34031)
0810 15:02:22.550153 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.550156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.550408 (+ 252us) server_negotiation.cc:407] Connection header received
0810 15:02:22.550708 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.550709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.550737 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.550767 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.554837 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.555594 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.557563 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.557879 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.557921 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.561811 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.561813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.561813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.561821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.563583 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.564757 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.564759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.564759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.564831 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.565473 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.565474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.565475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.565609 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.565682 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.566374 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.566549 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260810 15:02:22.569371 2339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837: Generated new TSK 0
I20260810 15:02:22.569602 2339 catalog_manager.cc:1514] Initializing in-progress tserver states...
Aug 10 15:02:22 dist-test-slave-cvwc krb5kdc[1937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374142, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.193.253@KRBTEST.COM for kudu/127.28.193.252@KRBTEST.COM
I20260810 15:02:22.576671 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.557184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.193.252:33861 (local address 127.0.0.1:54746)
0810 15:02:22.557274 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 15:02:22.557284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:02:22.557322 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:02:22.565975 (+ 8653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:02:22.565977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:02:22.566158 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:02:22.566208 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.566210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.567183 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.567185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.567980 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:02:22.567984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.568489 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.568490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:02:22.568547 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.568955 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:02:22.568960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:02:22.570462 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:02:22.575575 (+ 5113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.575576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.575576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.575834 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.576017 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:02:22.576018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:02:22.576018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:02:22.576094 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:02:22.576353 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:02:22.576353 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:02:22.576396 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:02:22.576467 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:02:22.576571 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 15:02:22.576682 2360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.557400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.28.193.252:33861)
0810 15:02:22.565799 (+ 8399us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.565801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.565812 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:02:22.565834 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.565835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.565847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.566288 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.566324 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.567080 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.568090 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.568396 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.568710 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.573770 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.573772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.573772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.573779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.575496 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.575903 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.575904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.575905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.575968 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.576148 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.576149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.576149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.576291 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.576349 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.576492 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.576571 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8373,"thread_start_us":89,"threads_started":1}
I20260810 15:02:22.578593 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: Bootstrap starting.
I20260810 15:02:22.581454 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: Neither blocks nor log segments found. Creating new log.
I20260810 15:02:22.581806 2165 log.cc:824] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: Log is configured to *not* fsync() on all Append() calls
I20260810 15:02:22.583086 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.550269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.28.193.252:33861)
0810 15:02:22.550310 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.550312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.568900 (+ 18588us) server_negotiation.cc:407] Connection header received
0810 15:02:22.569180 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.569181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.569206 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.569234 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.571673 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.572380 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.575842 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.576157 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.576197 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.579682 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.579684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.579684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.579691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.581182 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.581996 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.581998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.581998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.582061 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.582362 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.582363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.582364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.582526 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.582619 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.582915 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.582995 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:02:22.583657 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:22.550553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.28.193.253:46001)
0810 15:02:22.550626 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.550628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.562741 (+ 12113us) server_negotiation.cc:407] Connection header received
0810 15:02:22.563679 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.563680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.563695 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.563724 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.568046 (+ 4322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.568761 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.573089 (+ 4328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:22.573421 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:22.573472 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:22.578541 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:02:22.578543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:02:22.578543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:02:22.578550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:02:22.580619 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.582655 (+ 2036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.582656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.582656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.582723 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:02:22.582998 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:02:22.582999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:02:22.583000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:02:22.583133 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:02:22.583205 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:22.583490 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 15:02:22.583571 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:02:22.590444 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: No bootstrap required, opened a new log
I20260810 15:02:22.592546 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.592691 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d00045f915546a49680be435a64bd75, State: Initialized, Role: FOLLOWER
I20260810 15:02:22.592777 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } }
I20260810 15:02:22.593132 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:02:22.593235 2129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8d00045f915546a49680be435a64bd75"
I20260810 15:02:22.593326 2129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:02:22.594290 2129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4c1e6fa9ec04542a00e021b5f7c8837 in term 1.
I20260810 15:02:22.594512 2126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [term 1 FOLLOWER]: Refusing update from remote peer a4c1e6fa9ec04542a00e021b5f7c8837: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:02:22.593079 2364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.595337 2364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:22.596213 2357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:02:22.597693 2364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4c1e6fa9ec04542a00e021b5f7c8837. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.597801 2364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:02:22.597836 2364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:22.597879 2364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:02:22.598690 2385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:02:22.598786 2385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:02:22.599903 2387 mvcc.cc:205] Tried to move back new op lower bound from 7316988487957606400 to 7316988487851446272. Current Snapshot: MvccSnapshot[applied={T|T < 7316988487861055488}]
W20260810 15:02:22.614552 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.610021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.28.193.253:46001)
0810 15:02:22.610067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.610069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.613900 (+ 3831us) server_negotiation.cc:407] Connection header received
0810 15:02:22.614015 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.614016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.614027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.614055 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.614486 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53774: BlockingRecv error: recv got EOF from 127.0.0.1:53774 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 15:02:22.615293 2066 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.610290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.28.193.254:34031)
0810 15:02:22.610329 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.610330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.614786 (+ 4456us) server_negotiation.cc:407] Connection header received
0810 15:02:22.614878 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.614879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.614890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.614916 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.615232 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57070: BlockingRecv error: recv got EOF from 127.0.0.1:57070 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:02:22.615990 2289 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.610769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.28.193.252:33861)
0810 15:02:22.610807 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.610809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.615509 (+ 4700us) server_negotiation.cc:407] Connection header received
0810 15:02:22.615600 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.615601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.615611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.615638 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.615917 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54756: BlockingRecv error: recv got EOF from 127.0.0.1:54756 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:02:22.622836 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.622021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.28.193.253:46001)
0810 15:02:22.622061 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.622062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.622287 (+ 225us) server_negotiation.cc:407] Connection header received
0810 15:02:22.622389 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.622390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.622409 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.622436 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.622785 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53776: BlockingRecv error: recv got EOF from 127.0.0.1:53776 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:02:22.626046 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.625168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.28.193.253:46001)
0810 15:02:22.625202 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.625204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.625491 (+ 287us) server_negotiation.cc:407] Connection header received
0810 15:02:22.625589 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.625590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.625601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.625628 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.625998 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53782: BlockingRecv error: recv got EOF from 127.0.0.1:53782 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:02:22.629995 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.629154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.28.193.253:46001)
0810 15:02:22.629187 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.629188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.629455 (+ 267us) server_negotiation.cc:407] Connection header received
0810 15:02:22.629549 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.629550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.629559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.629586 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.629942 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53790: BlockingRecv error: recv got EOF from 127.0.0.1:53790 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 15:02:22.636097 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.635213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.28.193.253:46001)
0810 15:02:22.635261 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.635263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.635549 (+ 286us) server_negotiation.cc:407] Connection header received
0810 15:02:22.635646 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.635647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.635657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.635685 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.636051 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53800: BlockingRecv error: recv got EOF from 127.0.0.1:53800 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:02:22.646216 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.645316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.28.193.253:46001)
0810 15:02:22.645355 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.645356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.645604 (+ 248us) server_negotiation.cc:407] Connection header received
0810 15:02:22.645706 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.645708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.645751 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.645782 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.646164 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53814: BlockingRecv error: recv got EOF from 127.0.0.1:53814 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:02:22.664537 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.663588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.28.193.253:46001)
0810 15:02:22.663634 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.663637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.663920 (+ 283us) server_negotiation.cc:407] Connection header received
0810 15:02:22.664041 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.664043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.664056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.664084 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.664480 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50358: BlockingRecv error: recv got EOF from 127.0.0.1:50358 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:02:22.698611 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.697890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.28.193.253:46001)
0810 15:02:22.697935 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.697937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.698079 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:02:22.698135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.698136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.698148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.698198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.698551 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50372: BlockingRecv error: recv got EOF from 127.0.0.1:50372 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:02:22.764350 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.763707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.28.193.253:46001)
0810 15:02:22.763777 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.763780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.763829 (+ 49us) server_negotiation.cc:407] Connection header received
0810 15:02:22.763900 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.763901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.763912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.763962 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.764291 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50384: BlockingRecv error: recv got EOF from 127.0.0.1:50384 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 15:02:22.894369 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:22.893577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.28.193.253:46001)
0810 15:02:22.893628 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:22.893630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:22.893670 (+ 40us) server_negotiation.cc:407] Connection header received
0810 15:02:22.893848 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:22.893850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:22.893867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:22.893913 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:22.894307 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50388: BlockingRecv error: recv got EOF from 127.0.0.1:50388 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 15:02:23.152300 2172 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:02:23.151636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.28.193.253:46001)
0810 15:02:23.151696 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.151698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.151712 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:02:23.151820 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.151821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.151832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.151881 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:02:23.152235 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50390: BlockingRecv error: recv got EOF from 127.0.0.1:50390 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:02:23.221351 2357 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4c1e6fa9ec04542a00e021b5f7c8837" member_type: VOTER last_known_addr { host: "127.28.193.254" port: 34031 } } peers { permanent_uuid: "8d00045f915546a49680be435a64bd75" member_type: VOTER last_known_addr { host: "127.28.193.253" port: 46001 } } peers { permanent_uuid: "ade09fb5788449ff814ed955f296c902" member_type: VOTER last_known_addr { host: "127.28.193.252" port: 33861 } } }
I20260810 15:02:23.221508 2357 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4c1e6fa9ec04542a00e021b5f7c8837 [sys.catalog]: This master's current role is: LEADER
I20260810 15:02:23.519522 2307 catalog_manager.cc:1259] Loaded cluster ID: c486cef9643d477dbbfcbf37e194f999
I20260810 15:02:23.519600 2307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: loading cluster ID for follower catalog manager: success
I20260810 15:02:23.520418 2307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: acquiring CA information for follower catalog manager: success
I20260810 15:02:23.520707 2307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ade09fb5788449ff814ed955f296c902: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:02:23.599258 2385 catalog_manager.cc:1259] Loaded cluster ID: c486cef9643d477dbbfcbf37e194f999
I20260810 15:02:23.599341 2385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: loading cluster ID for follower catalog manager: success
I20260810 15:02:23.600111 2385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: acquiring CA information for follower catalog manager: success
I20260810 15:02:23.600378 2385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d00045f915546a49680be435a64bd75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:02:23.667969 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:23.665398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.28.193.253:46001)
0810 15:02:23.665514 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.665516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.665564 (+ 48us) server_negotiation.cc:407] Connection header received
0810 15:02:23.665645 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.665646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.665703 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.665797 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:23.665948 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.666439 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.667203 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.667396 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.667455 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:23.667611 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:23.667727 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:23.667796 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:23.667843 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:02:23.667864 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260810 15:02:23.671940 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:23.669621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.28.193.252:33861)
0810 15:02:23.669744 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.669746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.669755 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:02:23.669797 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.669797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.669833 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.669900 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:23.669973 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.670449 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.671180 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.671370 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.671432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:23.671586 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:23.671694 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:23.671752 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:23.671851 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 15:02:23.671868 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":55,"threads_started":1}
I20260810 15:02:23.676411 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:23.673766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.28.193.252:33861)
0810 15:02:23.673820 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.673822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.674102 (+ 280us) server_negotiation.cc:407] Connection header received
0810 15:02:23.674149 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.674150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.674160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.674205 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:23.674306 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.674968 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.675705 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.675983 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.676044 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:23.676166 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:23.676197 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:23.676246 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:23.676315 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:02:23.676336 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:02:23.676532 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:23.673523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.28.193.253:46001)
0810 15:02:23.673561 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.673563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.673655 (+ 92us) server_negotiation.cc:407] Connection header received
0810 15:02:23.673855 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.673856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.673867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.673924 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:23.674097 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.674643 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.675657 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.675881 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.675945 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:23.676243 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:23.676272 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:23.676315 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:23.676431 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:02:23.676457 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:02:23.676844 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:02:23.673578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.28.193.254:34031)
0810 15:02:23.673700 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 15:02:23.673702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:02:23.673728 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:02:23.673786 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:02:23.673787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:02:23.673844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:02:23.673925 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:02:23.674024 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.674785 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.675744 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:02:23.676049 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:02:23.676104 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:02:23.676280 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:02:23.676390 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:02:23.676726 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 15:02:23.676743 (+ 17us) server_negotiation.cc:299] Negotiation successful
0810 15:02:23.676769 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260810 15:02:23.678381 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 1954
I20260810 15:02:23.704677 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2069
I20260810 15:02:23.715370 29447 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskILwtmC/build/release/bin/kudu with pid 2176
2026-08-10T15:02:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2972 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5942 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274376 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST